End-to-end reinforcement learning for autonomous driving in urban environments

• Main Author: Coelho, Daniel Filipe Silveira
• Publication Date: 2024
• Language: eng
• Source: Repositórios Científicos de Acesso Aberto de Portugal (RCAAP)
• Download full: http://hdl.handle.net/10773/42846
• topic: Autonomous driving; Reinforcement learning; Deep learning; End-to-End systems

This thesis advances the field of end-to-end Autonomous Driving (AD) in urban environments, focusing primarily on Reinforcement Learning (RL) techniques to address existing limitations. The research begins with a comprehensive review of current end-to-end AD systems, highlighting the strengths and weaknesses of Imitation Learning (IL) and RL approaches, and identifying critical areas for improvement and promising avenues for future research. To address these challenges, we introduced RLAD, the first Reinforcement Learning from Pixels (RLfP) method for urban AD. Several techniques were introduced to enhance the performance of state-of-the-art methods. First, we developed an image encoder that utilizes both image augmentations and Adaptive Local Signal Mixing (A-LIX) layers. Additionally, we introduced WayConv1D, a waypoint encoder that captures the 2D geometrical information of waypoints using 1D convolutions. Furthermore, we designed an auxiliary loss function to emphasize the significance of traffic lights in the latent representation of the environment. RLAD demonstrated good performance on the NoCrash benchmark but required further integration of demonstrations to match state-of-the-art AD systems. Consequently, we developed RLfOLD (Reinforcement Learning from Online Demonstrations), which combines IL and RL by incorporating online demonstrations into RL training. We proposed a policy network that outputs two standard deviations, enabling adaptive control for exploration and IL training while considering uncertainty in both domains. Additionally, we incorporated an uncertainty-based technique guided by an online expert to enhance the exploration process. RLfOLD achieves state-of-the-art results on the NoCrash benchmark with enhanced efficiency and resource utilization. We further tackled the CARLA Leaderboard 2.0 benchmark’s challenges by developing PRIBOOT, an expert agent leveraging privileged information and limited human driving logs to navigate demanding scenarios. PRIBOOT’s novel techniques, such as the bird’s-eye view (BEV) representation and processing the BEV as an RGB image instead of a set of masks, significantly improved performance. This approach provided an expert capable of navigating this challenging benchmark, enabling researchers to generate extensive datasets and potentially resolving the data availability issues that have hindered progress. Collectively, our work presents significant contributions to the field of AD, offering insights and tools that pave the way for safer and more reliable autonomous systems in urban environments.