Enhancing Infrastructure Observability: Machine Learning for Proactive Monitoring and Anomaly Detection

• Main Author: Noetzold, Darlan
• Publication Date: 2024
• Other Authors: Rossetto, Anubis G. D. M., Leithardt, Valderi R. Q., Costa, Humberto J. de M.
• Format: Article
• Language: eng
• Source: Journal of internet services and applications (Internet)
• Download full: https://journals-sol.sbc.org.br/index.php/jisa/article/view/4509
• topic: Machine Learning; Infrastructure Monitoring; Anomaly Detection; Proactive Maintenance

This study addresses the critical challenge of proactive anomaly detection and efficient resource management in infrastructure observability. Introducing an innovative approach to infrastructure monitoring, this work integrates machine learning models into observability platforms to enhance real-time monitoring precision. Employing a microservices architecture, the proposed system facilitates swift and proactive anomaly detection, addressing the limitations of traditional monitoring methods that often fail to predict potential issues before they escalate. The core of this system lies in its predictive models that utilize Random Forest, Gradient Boosting, and Support Vector Machine algorithms to forecast crucial metric behaviors, such as CPU usage and memory allocation. The empirical results underscore the system's efficacy, with the GradientBoostingRegressor model achieving an R² score of 0.86 for predicting request rates, and the RandomForestRegressor model significantly reducing the Mean Squared Error by 2.06% for memory usage predictions compared to traditional monitoring methods. These findings not only demonstrate the potential of machine learning in enhancing observability but also pave the way for more resilient and adaptive infrastructure management.