White-box assessment for programming education

• Main Author: Caniço, Afonso Manuel Barral
• Publication Date: 2024
• Format: Master thesis
• Language: eng
• Source: Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
• Download full: http://hdl.handle.net/10071/32252
• topic: Programming education; White-box testing; Programming assessment; Feedback; Educação de programação; Testes de white-box; Avaliação de programação

Software testing is mostly performed in a black-box manner, that is, without incorporating any knowledge of the internal workings of programs into the tests. This practice usually suffices for enterprises and general practitioners, where the focus lies on producing reliable results while most algorithmic tasks are provided by third-party libraries. However, for computer science students and the like, it might not be straightforward to discern the underlying causes of an incorrect test result or to understand why certain algorithmic goals are not met. This dissertation presents Witter, a software testing library that allows programming instructors to define white-box tests for Java source code. Our tests analyse the execution of a method against a reference solution, to verify that the code not only produces correct results but is also in accordance with a desired algorithm behaviour. We detail the motivation for the development of the library and describe its development process in accordance with our research goals. We evaluate Witter’s efficacy in evaluating student submissions for an introductory university programming course to determine if an assessment tool based on code execution events could offer additional insight regarding incomplete algorithmic behaviour requirements, even when the student’s code produces correct outputs. The results support our hypothesis, and we conclude that a considerable amount of students make mistakes related to the algorithmic behaviour of their implemented solutions, and could thus benefit from an assessment tool providing them with information regarding their program’s execution.