System for capturing and displaying portuguese sign language animations in a 3D environment

• Autor(a) principal: Ribeiro, Bruno Filipe Matos
• Data de Publicação: 2024
• Tipo de documento: Dissertação
• Idioma: eng
• Título da fonte: Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
• Texto Completo: http://hdl.handle.net/20.500.11960/4029
• topic: Sign language; Translation; Avatar; Virtual interpreter; Motion capture; Glosses; Unity; Língua gestual; Tradução; Intérprete virtual; Captura de movimento; Glosa

Communication relies on a system of linguistic structure and rules that creates a common understanding between people. However, several issues, including language variation and cultural differences, can impact this process. A particular challenging context is the interaction between hearing and deaf individuals, since their methods of communication differ greatly. In Portugal, Portuguese Sign Language (LGP) is one of the three official languages, and it is spoken by around 110,000 people. This number is equivalent to around 1% of the population, so it is evident how difficult communication is for deaf people in Portugal. There are already several technologies that allow the translation between a wide range of spoken languages, but this is not yet the case between spoken and sign languages. This research project, which arises within the scope of the IVLinG project – Intérprete Virtual de Língua Gestual (Virtual Interpreter of Sign Language), aims to capture LGP gestures using Motion Capture (MoCap) devices and develop a mobile application to display those animations in a virtual environment, through a 3D Avatar. This part of the system is ready to receive a list of glosses, for example, after a Portuguese sentence is translated. The glosses received are used to determine what animations the Avatar shall animate. The first step was to conduct research on implementations on the same subject, specifically those that previously developed applications using LGP or other sign languages. Then, some LGP signs were captured as animations and an application was developed to receive the translator's glosses and animate the Avatar accordingly. The developed solution was tested to check if the translations could be done quickly and if the animations were understandable and correct according to LGP specialists. Tests carried out regarding the speed of the solution showed that, although the Avatar module takes a few seconds to load, after this step the animations are ready very quickly. Tests regarding the quality of the animations showed that MoCap can help improve the fluidity and naturalness of the animations, when comparing to manual animation, albeit several changes and refinements to the raw data are required. The work described in this document serves as a basis for a virtual translator from Portuguese to LGP and can significantly help the deaf community on their daily communication with the hearing community. Future work aims to improve the number of body and facial animations, using MoCap to assist in this process, and to improve the quality of these animations, by performing further testing to them and editing accordingly.