Significance extraction based on data augmentation for reinforcement learning

Abstract

Deep reinforcement learning has shown remarkable capabilities in visual tasks, but it does not have a good generalization ability in the context of interference signals in the input images; this approach is therefore hard to be applied to trained agents in a new environment. To enable agents to distinguish between noise signals and important pixels in images, data augmentation techniques and the establishment of auxiliary networks are proven effective solutions. We introduce a novel algorithm, namely, saliency-extracted Q-value by augmentation (SEQA), which encourages the agent to explore unknown states more comprehensively and focus its attention on important information. Specifically, SEQA masks out interfering features and extracts salient features and then updates the mask decoder network with critic losses to encourage the agent to focus on important features and make correct decisions. We evaluate our algorithm on the DeepMind Control generalization benchmark (DMControl-GB), and the experimental results show that our algorithm greatly improves training efficiency and stability. Meanwhile, our algorithm is superior to state-of-the-art reinforcement learning methods in terms of sample efficiency and generalization in most DMControl-GB tasks.

1. Introduction

Visual observation based reinforcement learning (RL) (Arulkumaran et al., 2017) has achieved tremendous success in various fields such as gaming (Mnih et al., 2013), robotic manipulation (Levine et al., 2016; Kalashnikov et al., 2018; Nair et al., 2018), autonomous navigation (Zhu et al., 2016; Yang W et al., 2019), and natural language processing (NLP) (Luketina et al., 2019). This learning approach continually optimizes decision-making processes by enabling the agent to interact with the environment to maximize long-term objectives. Despite excelling in specific virtual scenarios, RL algorithms face numerous challenges when applied in the real world. The complexity and variability of real-world environments mean that even a well-trained agent in one environment may struggle to adapt to new environments (Zhao et al., 2022). Research works indicate severe deficiencies in the generalization capability of agents (Farebrother et al., 2018; Cobbe et al., 2019; Gamrian and Goldberg, 2019; Song et al., 2020), as agents may fail to ignore noise factors and to focus attention on critical state features. Consequently, agents are unable to explore effectively in unknown environments (Henderson et al., 2017).

Recent research advancements have demonstrated that the generalization capability of RL models can be effectively enhanced through domain randomization (Tobin et al., 2017; Pinto et al., 2018) and data augmentation techniques (Yarats et al., 2021; Zhou, 2024). By increasing the randomness and diversity of the training data, these methods effectively help the agents better adapt to the new environment and thus enable the agents to try more states. Despite continuous efforts to improve the generalization performance of RL models (Hansen et al., 2021b; Yarats et al., 2021; Bertoin et al., 2022), existing algorithms still have shortcomings in focusing on important pixels and exploring the state space. These issues limit their effectiveness in real-world applications. Therefore, data augmentation techniques play a crucial role in driving the development of large-scale RL. Future research needs to further explore how to leverage these techniques more effectively such that agents can better understand and adapt to complex and dynamic real-world environments.