Fault Revealing Test Oracles, Are We There Yet? Evaluating The Effectiveness Of Automatically Generated Test Oracles On Manually-Written And Automatically Generated Unit Tests

Bibliographic Details
Main Author: Bento, Daniel Correia
Publication Date: 2022
Format: Master thesis
Language: eng
Source: Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
Download full: http://hdl.handle.net/10451/56845
Summary: Tese de mestrado, Engenharia Informática, 2022, Universidade de Lisboa, Faculdade de Ciências
id RCAP_a1f77337af28a73e1ae0a29fb9963ca7
oai_identifier_str oai:repositorio.ulisboa.pt:10451/56845
network_acronym_str RCAP
network_name_str Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
repository_id_str https://opendoar.ac.uk/repository/7160
spelling Fault Revealing Test Oracles, Are We There Yet? Evaluating The Effectiveness Of Automatically Generated Test Oracles On Manually-Written And Automatically Generated Unit TestsTeste de softwareTestes unitáriosTest oracleEstudo empíricoGeração automática de oraclesTeses de mestrado - 2023Departamento de InformáticaTese de mestrado, Engenharia Informática, 2022, Universidade de Lisboa, Faculdade de CiênciasAutomated test suite generation tools have been used in real development scenarios and proven to be able to detect real faults. These tools, however, do not know the expected behavior of the system and generate tests that execute the faulty behavior, but fail to identify the fault due to poor test oracles. To solve this problem, researchers have developed several approaches to automatically generate test oracles that resemble manually-written ones. However, there remain some questions regarding the use of these tools in real development scenarios. In particular, how effective are automatically generated test oracles at revealing real faults? How long do these tools require to generate an oracle? To answer these questions, we applied a recent and promising test oracle generation approach (T5) to all fault-revealing test cases in the DEFECTS4J collection and investigated how effective are the generated test oracles at detecting real faults as well as the time required by the tool to generate them; Our results show that: (1) out-of-the-box, oracles generated by T5 do not compile; (2) after a simple procedure, out of the 1696 test oracles, only 466 compile and 58 of them manage to correctly identify the fault; (3) when considering the 835 bugs in DEFECTS4J, T5 was able to detect 27, i.e., 3.23% of the bugs. Moreover, T5 required, on average, 401.3 seconds to generate a test oracle. The approaches and datasets presented in this thesis bring automated test oracle generation one step closer to being used in real software, by providing insight into current problems of several tools as well as introducing a way to test automated test oracle generation tools that are being developed regarding their effectiveness on detecting real software faults.Campos, José Carlos Medeiros deRepositório da Universidade de LisboaBento, Daniel Correia2023-03-28T10:07:19Z202320222023-01-01T00:00:00Zinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/masterThesisapplication/pdfhttp://hdl.handle.net/10451/56845TID:203497287enginfo:eu-repo/semantics/openAccessreponame:Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)instname:FCCN, serviços digitais da FCT – Fundação para a Ciência e a Tecnologiainstacron:RCAAP2025-03-17T14:55:34Zoai:repositorio.ulisboa.pt:10451/56845Portal AgregadorONGhttps://www.rcaap.pt/oai/openaireinfo@rcaap.ptopendoar:https://opendoar.ac.uk/repository/71602025-05-29T03:29:06.953490Repositórios Científicos de Acesso Aberto de Portugal (RCAAP) - FCCN, serviços digitais da FCT – Fundação para a Ciência e a Tecnologiafalse
dc.title.none.fl_str_mv Fault Revealing Test Oracles, Are We There Yet? Evaluating The Effectiveness Of Automatically Generated Test Oracles On Manually-Written And Automatically Generated Unit Tests
title Fault Revealing Test Oracles, Are We There Yet? Evaluating The Effectiveness Of Automatically Generated Test Oracles On Manually-Written And Automatically Generated Unit Tests
spellingShingle Fault Revealing Test Oracles, Are We There Yet? Evaluating The Effectiveness Of Automatically Generated Test Oracles On Manually-Written And Automatically Generated Unit Tests
Bento, Daniel Correia
Teste de software
Testes unitários
Test oracle
Estudo empírico
Geração automática de oracles
Teses de mestrado - 2023
Departamento de Informática
title_short Fault Revealing Test Oracles, Are We There Yet? Evaluating The Effectiveness Of Automatically Generated Test Oracles On Manually-Written And Automatically Generated Unit Tests
title_full Fault Revealing Test Oracles, Are We There Yet? Evaluating The Effectiveness Of Automatically Generated Test Oracles On Manually-Written And Automatically Generated Unit Tests
title_fullStr Fault Revealing Test Oracles, Are We There Yet? Evaluating The Effectiveness Of Automatically Generated Test Oracles On Manually-Written And Automatically Generated Unit Tests
title_full_unstemmed Fault Revealing Test Oracles, Are We There Yet? Evaluating The Effectiveness Of Automatically Generated Test Oracles On Manually-Written And Automatically Generated Unit Tests
title_sort Fault Revealing Test Oracles, Are We There Yet? Evaluating The Effectiveness Of Automatically Generated Test Oracles On Manually-Written And Automatically Generated Unit Tests
author Bento, Daniel Correia
author_facet Bento, Daniel Correia
author_role author
dc.contributor.none.fl_str_mv Campos, José Carlos Medeiros de
Repositório da Universidade de Lisboa
dc.contributor.author.fl_str_mv Bento, Daniel Correia
dc.subject.por.fl_str_mv Teste de software
Testes unitários
Test oracle
Estudo empírico
Geração automática de oracles
Teses de mestrado - 2023
Departamento de Informática
topic Teste de software
Testes unitários
Test oracle
Estudo empírico
Geração automática de oracles
Teses de mestrado - 2023
Departamento de Informática
description Tese de mestrado, Engenharia Informática, 2022, Universidade de Lisboa, Faculdade de Ciências
publishDate 2022
dc.date.none.fl_str_mv 2022
2023-03-28T10:07:19Z
2023
2023-01-01T00:00:00Z
dc.type.status.fl_str_mv info:eu-repo/semantics/publishedVersion
dc.type.driver.fl_str_mv info:eu-repo/semantics/masterThesis
format masterThesis
status_str publishedVersion
dc.identifier.uri.fl_str_mv http://hdl.handle.net/10451/56845
TID:203497287
url http://hdl.handle.net/10451/56845
identifier_str_mv TID:203497287
dc.language.iso.fl_str_mv eng
language eng
dc.rights.driver.fl_str_mv info:eu-repo/semantics/openAccess
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.source.none.fl_str_mv reponame:Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
instname:FCCN, serviços digitais da FCT – Fundação para a Ciência e a Tecnologia
instacron:RCAAP
instname_str FCCN, serviços digitais da FCT – Fundação para a Ciência e a Tecnologia
instacron_str RCAAP
institution RCAAP
reponame_str Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
collection Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
repository.name.fl_str_mv Repositórios Científicos de Acesso Aberto de Portugal (RCAAP) - FCCN, serviços digitais da FCT – Fundação para a Ciência e a Tecnologia
repository.mail.fl_str_mv info@rcaap.pt
_version_ 1833601715180404736

Similar Items