Gesture recognition for interaction tasks in immersive reality

• Main Author: Santos, José Luís Picão
• Publication Date: 2024
• Format: Master thesis
• Language: eng
• Source: Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
• Download full: http://hdl.handle.net/10400.26/54137
• topic: Computer vision; Pose tracking; Skeleton tracking; Gesture classification; Immersive reality

With continuous advancements in Immersive technology, enhancing humancomputer interaction (HCI) with virtual environments has become a relevant topic. Commercial virtual reality (VR) systems are equipped with handheld controllers, which are limited in naturalness and user interactions. Immersive technologies benefit from the sense of presence, immersion, and embodiment to augment user experience. This project addresses these aspects by developing a computer vision-based dynamic gesture recognition system with full-body pose tracking integrated into a virtual reality experience. Unlike most existing applications that use only body tracking, our system combines both body and hand tracking and includes a gesture recognition module. To demonstrate the effectiveness of the approach, we propose a 3D virtual immersive environment scenario where the user engages in a game of "rock-paper-scissors" against the system, aiming to outperform it. This project uses the Mediapipe framework as a body pose tracking mechanism to extract the user’s articulation coordinates. The obtained data is processed using a long-short-term memory (LSTM) deep neural network (DNN) to classify dynamic gestures. The game engine Unity 3D is used to represent the avatar and to present the immersive experience to the user. To validate the developed system, experimental tasks were conducted with eight participants. Each participant had to play the game 5 times. The system was validated quantitatively by measuring the online classification accuracy, and the v subjective sense of realism, presence, involvement and system usability were evaluated using the iGroup Presence Questionnaire (IPQ). Results show the effective possibility of tracking both body and hands, with potential applications ranging from rehabilitation to sports. Regarding the integration of pose tracking for controlling the avatar in the 3D immersive VR environment, results indicate the experience causes a positive impact on users in terms of realism and presence, as evidenced by a 73.44% score on the IPQ. However, users reported a mismatch in their movements and the avatars during the experience. The performance of the gesture recognition classification model did not match the accuracy achieved during the offline validation and testing phases. This lack of generalisation of the model is attributed to the limited number of training samples and the low variability in participant's gestures, as the dataset included only a single individual. Overall, the biggest challenge of the project was the integration of the pose data to control the avatar. Nevertheless, the results demonstrated the feasibility of combining computer vision-based gesture interaction and full-body tracking ina VR experience