Data engineering pipeline as a service for MLOps initiatives

• Main Author: Fazenda, Miguel Filipe Rodrigues Almeida de Matos
• Publication Date: 2023
• Format: Master thesis
• Language: eng
• Source: Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
• Download full: http://hdl.handle.net/10773/41929
• topic: AWS; Big data; Data engineering; DevOps; Docker; GitOps; MLOps; Pipeline; Kubernetes

The companies of today increasingly need to use data to ground their decisions with crucial information. To make the best use of it, data needs to be processed and analyzed for information to be extracted from it. However, information extraction from data can be a long and complex process, due to the possibility of data having enormous volumes. The handling of large volumes of data represents the concept of Big Data To stay ahead of the competition, companies need to use systems designed according to Data Engineering principles in order to handle these large volumes of data. Data Engineering is a discipline that focuses on the construction of systems that can ingest, process, and store large amounts of data. The objective of this dissertation is the construction of a system, more precisely a pipeline, that can handle large volumes of data, related to electronic products, and apply ML models on top of it to predict the next value of the intended product. The predicted values should then be stored and served to users. The system has some limitations imposed regarding the architecture and tooling, it must be based on microservices, cloud-agnostic, containerized, orchestrated, based on the SMACK stack, and use free and open-source tools. This system serves as an alternative for MLOps startups, which combine Data Engineering, DevOps, and ML to process data. The development of the system will be done with the GitOps operational framework, which applies DevOps best practices, such as versioning, compliance, collaboration, and CI/CD, and applies them to infrastructure automation. The system passed through multiple iterations, each representing a stage of the development. An initial iteration made in Docker Compose to serve as Proof of Concept, a middle one to adapt the pipeline to the Kubernetes environment and test it, and a final one on AWS through EKS, representing a real-life production scenario. The Kubernetes versions have monitoring in order to facilitate testing and observation of the system. In general, this document approaches the tools chosen, the multiple versions of the pipeline and objectives of each, and the obtained results and meaning behind them.