ETL for data science?: A case study

• Autor(a) principal: Oliveira, Nicole Furtado
• Data de Publicação: 2021
• Tipo de documento: Dissertação
• Idioma: eng
• Título da fonte: Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
• Texto Completo: http://hdl.handle.net/10071/23699
• topic: Data mining --; Business understanding; Data understanding; Data preparation; Data extraction; Data transformation; Data cleaning; Data normalization; Pipeline; ETL; Data warehouse; Process models; Methodologies; CRISP-DM; KDD; Python; Grant funds management; Grant funds for the private sector; European structural and investment funds; Machine learning; Artificial intelligence; Data science; Case study; Compreensão do negócio; Compreensão de dados; Preparação de dados; Extração de dados; Transformação de dados; Limpeza de dados; Normalização de dados; Modelos de processo; Metodologia; Gestão de fundos de investimento; Fundos de investimento para o setor privado; Fundos estruturais e de investimento europeus; Inteligência artificial; Ciência de dados; Caso de estudo

Big data has driven data science development and research over the last years. However, there is a problem - most of the data science projects don't make it to production. This can happen because many data scientists don’t use a reference data science methodology. Another aggravating element is data itself, its quality and processing. The problem can be mitigated through research, progress and case studies documentation about the topic, fostering knowledge dissemination and reuse. Namely, data mining can benefit from other mature fields’ knowledge that explores similar matters, like data warehousing. To address the problem, this dissertation performs a case study about the project “IA-SI - Artificial Intelligence in Incentives Management”, which aims to improve the management of European grant funds through data mining. The key contributions of this study, to the academia and to the project’s development and success are: (1) A combined process model of the most used data mining process models and their tasks, extended with the ETL’s subsystems and other selected data warehousing best practices. (2) Application of this combined process model to the project and all its documentation. (3) Contribution to the project’s prototype implementation, regarding the data understanding and data preparation tasks. This study concludes that CRISP-DM is still a reference, as it includes all the other data mining process models’ tasks and detailed descriptions, and that its combination with the data warehousing best practices is useful to the project IA-SI and potentially to other data mining projects.