Optimization methods in fully cooperative scenarios: a review of multiagent reinforcement learning

Abstract

Multiagent reinforcement learning (MARL) has become a dazzling new star in the field of reinforcement learning in recent years, demonstrating its immense potential across many application scenarios. The reward function directs agents to explore their environments and make optimal decisions within them by establishing evaluation criteria and feedback mechanisms. Concurrently, cooperative objectives at the macro level provide a trajectory for agents’ learning, ensuring alignment between individual behavioral strategies and the overarching system goals. The interplay between reward structures and cooperative objectives not only bolsters the effectiveness of individual agents but also fosters interagent collaboration, offering both momentum and direction for the development of swarm intelligence and the harmonious operation of multiagent systems. This review delves deeply into the methods for designing reward structures and optimizing cooperative objectives in MARL, along with the most recent scientific advancements in this field. The article meticulously reviews the application of simulation environments in cooperative scenarios and discusses future trends and potential research directions in the field, providing a forward-looking perspective and inspiration for subsequent research efforts.

1. Introduction

Multiagent reinforcement learning (MARL) has become a research hotspot in the field of reinforcement learning (RL) in recent years and has made significant progress, opening up a series of highly complex and challenging application areas, such as autonomous driving (Zhang KQ et al., 2021; Huang et al., 2024; Ren FY et al., 2024; Ren Y et al., 2024), drone collaboration (Jia et al., 2023; Nian et al., 2024; Wang BL et al., 2024), smart cities (Wu T et al., 2020; Qiao et al., 2024), smart grids (Xu X et al., 2020; Wang JH et al., 2021a; Gou et al., 2022), and robotic cooperation (Chen HB et al., 2023; Gu et al., 2023).

The reward function is a fundamental component of MARL, which guides agents to learn how to make optimal decisions within an environment by defining and providing a feedback mechanism (Icarte et al., 2022). The reward function operates by assessing the outcomes of agents’ actions and determining the appropriate rewards or penalties to be assigned. This feedback mechanism directly influences the agents’ behavioral strategies, as they learn to prioritize actions that yield higher rewards and avoid those that lead to penalties. In the domain of multiagent systems (MASs), crafting an appropriate reward structure is paramount, as it necessitates judicious reward allocation among individual agents and collective entities within the team. This is done to foster various interactive dynamics, such as cooperation or competition, and to guarantee the alignment of agent objectives toward the attainment of shared goals (Shou and Di, 2020). A meticulously formulated reward function is instrumental in augmenting the efficiency of collective learning processes, optimizing synergistic collaboration among agents, and equipping the agents with the capacity for adaptation and proficiency in environments characterized by their complexity and dynamism.

The optimization of cooperative objectives plays a crucial role within the framework of MARL, as it integrates and optimizes individual reward signals, providing a quantified optimization criterion for the adjustment of agent strategies. By aiming for the maximization of long-term cumulative rewards, it guides agents to search for the optimal sequence of actions within the strategy space and ensures the alignment of the agents’ decision-making processes with collective goals (Yang NK et al., 2023). This optimization not only reinforces the goal-directedness of agents but also promotes efficient collaboration and the development of adaptability within MASs.