Web scraping and analysis of car data

• Autor(a) principal: Silva, João Luís Magalhães da
• Data de Publicação: 2024
• Tipo de documento: Dissertação
• Idioma: eng
• Título da fonte: Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
• Texto Completo: http://hdl.handle.net/10400.22/26596
• topic: Web scraping; Data analysis; Application; Data extraction; Data insertion; Python libraries; Car data; Agregamento de dados; Análise de dados; Aplicação; Extração de dados; Inserção de dados; Bibliotecas Python; Dados de automóveis

The growth of online car marketplaces has created challenges in efficiently gathering and analyzing car data due to price fluctuations and increasing digital reliance. This thesis tackles the problem through web scraping and data analysis to assist in market insights. A review of web scraping tools like BeautifulSoup, Requests, and Selenium, alongside data analysis libraries such as Pandas, was conducted. A system was developed to scrape car data from Standvirtual and analyze key attributes like price and mileage. The data was processed using Python tools, and a Flask-based server application was built for easy access, with offline analysis supported through Excel. Challenges such as incomplete data and anti-scraping measures were resolved with advanced extraction techniques and error handling. Further improvements include optimizing the scraping process and integrating machine learning models for more accurate price predictions. In conclusion, the project demonstrates the potential of web scraping for car market analysis, providing a foundation for future predictive analytics and real-time data applications.