Reward shaping for goal-oriented tasks using deep reinforcement learning

Abstract

This thesis addresses the challenge of designing effective reward functions for training agents using deep reinforcement learning (DRL) in goal-oriented tasks. The main objective is to accelerate learning, minimize the occurrence of local optima, increase efficiency, and ensure that the agent correctly aligns with the defined objectives. The study explores how reward functions contribute to improving policy generalization, enabling robust performance in new and untrained scenarios. A key focus of this research is reducing the gap between simulation and reality, facilitating the transfer of policies learned in constrained simulated environments to complex and dynamic real-world applications without performance loss or the need for additional training. As a solution to the explored problem, we propose a reward function with reward shaping based on the Potential-Based Reward Shaping (PBRS) principle, where the reward is defined by the difference between potential functions. We demonstrate that adding the proposed reward shaping to the reward function does not interfere with policy optimality, ensuring that the agent learns the desired behavior while benefiting from the new reward function. The proposed reward function, applied to training agents in mapless autonomous robot navigation, achieves superior performance compared to other reward shaping functions in the literature that rely on distance to the target. Comparisons with existing reward functions indicate accelerated training convergence and an increase in the number of tasks completed in a test environment after training. Furthermore, the proposed method demonstrates robustness and achieves superior results compared to others in the literature when operating in environments different from those used for training. Similar results were obtained in zero-shot transfer, both in sim-to-sim and sim-to-real scenarios, outperforming existing methods even when transferred to robots with architectures different from those used during training. Thus, this innovative approach to reward shaping in robot navigation enhances generalization and its application in unknown scenarios, with promising results in both simulation and real-world environments.