Processo de desenvolvimento de uma feature store a partir de dados de web scraping

• Main Author: Silva, Pedro Augutos de Lima e
• Publication Date: 2023
• Format: Bachelor thesis
• Language: por
• Source: Repositório Institucional da UTFPR (da Universidade Tecnológica Federal do Paraná (RIUT))
• Download full: http://repositorio.utfpr.edu.br/jspui/handle/1/35499
• topic: Sistemas de coleta automática de dados; Mineração de dados (Computação); Armazenamento de dados; Processamento eletrônico de dados; Python (Linguagem de programação de computador); Estruturas de dados (Computação); Automatic data collection systems; Data mining; Data Warehousing; Electronic data processing; Python (Computer program language); Data structures (Computer science); CNPQ::CIENCIAS EXATAS E DA TERRA

With the growth of the electronic games industry and the dependence on data for decision making, the work seeks to present a data transformation process until it is significantly safe and ready for analysis or to be acquired by an machine model learning . As an example, let’s use event data from Counter Strike championship matches. This process consists of an web scraping step, where it was necessary to develop an script in Python to standardize the process, and another data transformation step using the Spark processing engine, since the amount of extracted data was considerably large. The result is an feature store, which is a set of data about the statistics of professional players in championships with prizes above 100 thousand dollars.