Integrating flakiness detection and repair

• Main Author: Mendes, Alexandre Filipe de Freitas
• Publication Date: 2024
• Format: Master thesis
• Language: eng
• Source: Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
• Download full: http://hdl.handle.net/10400.22/26427
• topic: Flaky tests; Non-determinism; Software testing; Order dependency; Testes flaky; Testes de software; Não-determinismo; Dependência de ordem

Testing is a core component of software development, safeguarding the reliability of the code. However, some tests may provide unexpected outcomes, non-deterministically passing or failing. Such tests are deemed flaky. Known to delay releases and reduce the overall effectiveness and efficiency of testing these tests have become a key problem. Affecting the entire industry, awareness, and research on the topic have risen significantly in recent years. This dissertation presents a systematic literature review on flaky test research's state of the art. It details the research methodology used and provides an overview of recent developments in the field. The review addresses four dimensions: causes, detection, repair, and tool integration. Each dimension is analyzed to classify flakiness and detail tools and methodologies capable of addressing different aspects of flakiness. In this manner, the dissertation comprehensively examines recent strategies for addressing flaky tests. Among the categories of flakiness, order-dependency has received the most research focus, with several tools available for detecting and repairing this category specifically. However, there is a notable lack of tools capable of both detecting and repairing flaky tests in a unified manner. By studying and evaluating individual tools, this dissertation assesses the potential compatibility between already existing and available methodologies, allowing for the development of a prototype for a unified order-dependency flakiness solution. Evaluating this prototype against the individual execution of its components reveals that integrating existing tools enhances usability while preserving comparable levels of recall and execution time. This analysis highlights the necessity for standardizing the inputs and outputs of order-dependency detection and repair tools. Additionally, it shows that complete detection