Γ-IRT : an item response theory model for evaluating regression algorithms

• Ano de defesa: 2021
• Autor(a) principal: MORAES, João Victor Campos
• Orientador(a): PRUDÊNCIO, Ricardo Bastos Cavalcante
• Tipo de documento: Dissertação
• Tipo de acesso: Acesso aberto
• Idioma: eng
• Instituição de defesa: Universidade Federal de Pernambuco
• Programa de Pós-Graduação: Programa de Pos Graduacao em Ciencia da Computacao
• País: Brasil
• Palavras-chave em Português: Inteligência artificial; Aprendizagem de máquina
• Link de acesso: https://repositorio.ufpe.br/handle/123456789/50976

Item Response Theory (IRT) is used to measure latent abilities of human respondents based on their responses to items with different difficulty levels. Recently, IRT has been applied to algorithm evaluation in Artificial Inteligence (AI), by treating the algorithms as respondents and the AI tasks as items. The most common models in IRT only deal with dichotomous responses (i.e., a response has to be either correct or incorrect). Hence they are not adequate in application contexts where responses are recorded in a continuous scale. In this dissertation we propose the Γ-IRT model, particularly designed for dealing with positive unbounded responses, which we model using a Gamma distribution, parameterised according to respondent ability and item difficulty and discrimination parameters. The proposed parameterisation results in item characteristic curves with more flexible shapes compared to the traditional logistic curves adopted in IRT. We apply the proposed model to assess regression model abilities, where responses are the absolute errors in test instances. This novel application represents an alternative for evaluating regression performance and for identifying regions in a regression dataset that present different levels of difficulty and discrimination.