Translation error annotation : building an annotation module for east asian languages

• Main Author: Silva, Beatriz Barrote
• Publication Date: 2022
• Format: Master thesis
• Language: eng
• Source: Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
• Download full: http://hdl.handle.net/10451/56071
• topic: Domínio/Área Científica::Humanidades::Línguas e Literaturas

In this thesis it is proposed an annotation module to be applied in the context of Machine Translation (MT) concerning the East Asian languages of Japanese, Korean and Mandarin for the purpose of assessing MT output quality through annotation. The annotation module was created based on a data-driven analysis over Customer Support content in these languages previously annotated with the Unbabel Error Typology, which is a general typology in the sense that it is not conceived for any specific groups of languages. As such, this work also explores how applying translation error typologies inadequate to certain languages or content types can have an impact on how annotation reflects the quality of a translation. For the purpose of testing the effectiveness of the proposed annotation module, an annotation experiment for the languages under analysis was conducted. This experiment consisted of, for each language, annotating the same content using three different error typologies: the Unbabel Error Typology, the MQM-compliant error taxonomy for the translation direction of English to Chinese proposed by Ye and Toral (2020) and the annotation module proposed on this thesis. Furthermore, each dataset was annotated by two annotators. This allowed a comparison of Inter-annotator agreement (IAA) scores, which constitutes an important metric in terms of evaluating the effectiveness of an error typology. In light of this, each of the tested typologies was analyzed based on the obtained IAA scores and a further in-depth analysis of concrete annotations which lead to an understanding over their strengths and limitations. With this work it was possible to demonstrate that, if on one hand using error typologies inadequate for the content annotated has a negative impact on the quality of said annotations, on the other hand applying an error typology specific to the content being annotated can result in more consistent annotations.