Reinforcement learning based privacy-preserving consensus tracking control of nonstrict-feedback discrete-time multi-agent systems

Abstract

This paper investigates a privacy-preserving consensus tracking problem for a class of nonstrict-feedback discrete-time multi-agent systems (MASs). An improved Liu cryptosystem is developed to alleviate the errors between encryption and decryption on the plaintext, which ensures satisfactory recovery of the plaintext information. A reinforcement learning (RL) technique is then employed to compensate for unknown dynamics and errors between true signals and decrypted ones. Based on the backstepping and graph theory, an RL-based privacy-preserving consensus tracking control strategy is further designed. By virtue of graph theory and Lyapunov stability theory, it is shown that the consensus tracking errors and all signals in the MAS are ultimately bounded. Finally, simulation examples are presented for verification of the effectiveness of the control strategy.

1. Introduction

In recent years, consensus control of multi-agent systems (MASs) has attracted extensive attention in the control community (Zhang HG et al., 2017; Li HY et al., 2021; He et al., 2022; Ju et al., 2022; Wen and Li, 2022; Yang XD et al., 2022; Li JN et al., 2023; Zhang XM et al., 2023). This is primarily because consensus control finds its broad applications in diverse areas, such as multiple autonomous vehicles (Shahvali et al., 2018; Peng et al., 2021; Ge XH et al., 2022, 2023, 2024; Xie et al., 2022), manipulators (Yu et al., 2019), aerospace engineering (Sakthivel et al., 2019), mobile robots (Ding et al., 2020; Ning et al., 2023), and power systems (Li P et al., 2022). Although MASs provide flexible and convenient platforms for industry and military applications, they are vulnerable to sensitive information disclosure due to their openness over shared networks. Research on privacy protection of MASs is thus of great significance (Nozari et al., 2017; Gao L et al., 2019; Kishida, 2019; Ruan et al., 2019; Wang YQ, 2019; Yin et al., 2020; Fang et al., 2021; Wang YQ et al., 2021; Xu et al., 2021; Liang et al., 2022; Chen et al., 2023).

Many techniques have been developed for preserving privacy in consensus control, such as homomorphic encryption (Fang et al., 2021; Wang YQ et al., 2021; Chen et al., 2023), differential privacy (Yang ZW et al., 2022), and state decomposition (Wang YQ, 2019). Specifically, Chen et al. (2023) addressed a privacy-preserving economic dispatch issue for microgrids by a homomorphically encrypted consensus strategy. Yan et al. (2021) and Wang AJ et al. (2022) offered distributed privacy protection algorithms and consensus-based economic dispatch strategies, ensuring information security and node privacy. In Fang et al. (2021), a homomorphic encryption based protocol was generalized for a second-order system. In Wang YQ et al. (2021), an agent decomposition strategy was introduced via Paillier encryption, and the security of agents’ initial states was analyzed. It is worth pointing out that these results do not consider the impact of privacy-preserving errors in an MAS, and these errors are of paramount importance in consensus behaviors.

Reinforcement learning (RL) plays a vital role in consensus control. A common architecture of RL is a critic–actor form. With the actor–critic architecture, state-feedback and output-feedback optimal control methods were proposed in Tong et al. (2018) and Li YM et al. (2020a, 2022) under the framework of backstepping technique. Li YM et al. (2020a) investigated an adaptive fuzzy inverse optimal control problem and developed an inverse optimal scheme. To address the issue of adaptive control of large-scale systems in strict-feedback form, a feedforward decentralized controller with adaptive laws was designed in Tong et al. (2018) using a state observer with the backstepping design technique. These advances motivate the integration of RL with privacy-preserving mechanisms for discrete-time nonstrict-feedback multi-agent systems.