Document Type : Original/Review Paper
Authors
Department of Electrical and Computer Engineering, Semnan University, Semnan, Iran.
Abstract
This paper introduces an adaptive optimal distributed algorithm based on event-triggered control to solve multi-agent discrete-time zero-sum graphical games for unknown nonlinear constrained-input systems with external disturbances. Based on the value iteration heuristic dynamic programming, the proposed algorithm solves the event-triggered coupled Hamilton-Jacobi-Isaacs equations assuming unknown dynamics to develop distributed optimal controllers and satisfy leader-follower consensus for agents interacting on a communication graph. The algorithm is implemented using the actor-critic neural network, and unknown system dynamics are approximated using the identifier network. Introducing and solving nonlinear zero-sum discrete-time graphical games in the presence of unknown dynamics, control input constraints and external disturbances, differentiate this paper from the previously published works. Also, the control input, external disturbance, and the neural network's weights are updated aperiodic and only at the triggering instants to simplify the computational process. The closed-loop system stability and convergence to the Nash equilibrium are proven. Finally, simulation results are presented to confirm theoretical findings.
Keywords
- Adaptive optimal control
- Event-triggering scheme
- Optimal leader-follower consensus
- Reinforcement learning
- Neural networks
Main Subjects
[1] D. L. Fernandes, A. L. M. Leopoldino, J. de Santiago, C. Verginis, A. A. Ferreira, and J. G. de Oliveira, “Distributed control on a multi-agent environment co-simulation for DC bus voltage control,” Electric Power Systems Research, vol. 232, pp. 110408, 2024.
[2] K. Sun, H. Yu, and X. Xia, “Distributed control of nonlinear stochastic multi-agent systems with external disturbance and time-delay via event-triggered strategy,” Neurocomputing, vol. 452, pp. 275-283, 2021.
[3] M. H. Rezaei and M. B. Menhaj, “Adaptive output stationary average consensus for heterogeneous unknown linear multi-agent systems,” IET Control Theory & Applications, vol. 12, no. 7, pp. 847-856, 2018.
[4] W. Meng, Q. Yang, J. Sarangapani, and Y. Sun, “Distributed control of nonlinear multiagent systems with asymptotic consensus,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 47, no. 5, pp. 749–757, May 2017.
[5] C. J. Li and G. P. Liu, “Data‐driven consensus for non‐linear networked multi‐agent systems with switching topology and time‐varying delays,” IET Control Theory & Applications, vol. 12, no. 12, pp. 1773–1779, Aug. 2018.
[6] H. Li, Q. Liu, G. Feng, and X. Zhang, “Leader–follower consensus of nonlinear time-delay multiagent systems: A time-varying gain approach,” Automatica, vol. 126, pp. 109444, Apr. 2021.
[7] J. Kandasamy, R. Ramachandran, V. Veerasamy, and A. X. R. Irudayaraj, “Distributed leader-follower based adaptive consensus control for networked microgrids,” Applied Energy, vol. 353, pp. 122083, 2024.
[8] F. Tatari, and M. B. Naghibi-Sistani, “Optimal adaptive leader-follower consensus of linear multi-agent systems: Known and unknown dynamics,” Journal of Artificial Intelligence and Data Mining, vol. 3, no. 1, pp. 101-111, 2015.
[9] J. Long, W. Wang, J. Huang, J. Lu, and K. Liu, “Adaptive leaderless consensus for uncertain high-order nonlinear multiagent systems with event-triggered communication,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 52, no. 11, pp. 7101–7111, Nov. 2022.
[10] M. Rehan, M. Tufail, and S. Ahmed, “Leaderless consensus control of nonlinear multi-agent systems under directed topologies subject to input saturation using adaptive event-triggered mechanism,” Journal of the Franklin Institute, vol. 358, no. 12, pp. 6217–6239, Aug. 2021.
[11] X. Liu, J. Sun, L. Dou, and J. Chen, “Leader-following consensus for discrete-time multi-agent systems with parameter uncertainties based on the event-triggered strategy,” Journal of Systems Science and Complexity, vol. 30, no. 1, pp. 30–45, Feb. 2017.
[12] Q. Yuan, G. Chen, Y. Tian, Y. Yuan, Q. Zhang, X. Wang, and J. Liu, “Leader-following consensus of discrete-time nonlinear multi-agent systems with asymmetric saturation impulsive control,” Mathematics, vol. 12, no. 3, pp. 469, 2024.
[13] T. Basar and G. J. Olsder, “Dynamic noncooperative game theory,” Society for Industrial and Applied Mathematics, 2nd ed, Jan. 1998.
[14] K. G. Vamvoudakis, F. L. Lewis, and G. R Hudas, “Multi-agent differential graphical games: Online adaptive learning solution for synchronization with optimality,” Automatica, vol. 48, no. 8, pp. 1598–1611, Aug. 2012.
[15] Y. Guo, Q. Sun, Y. Wang, and Q. Pan, “Differential graphical game‐based multi‐agent tracking control using integral reinforcement learning,” IET Control Theory & Applications, 2024.
[16] F. Tatari, M. B. Naghibi-Sistani, and K. G. Vamvoudakis, “Distributed optimal synchronization control of linear networked systems under unknown dynamics,” American Control Conference(ACC), pp. 668-673, IEEE , May 2017.
[17] F. Tatari, M. B. Naghibi-Sistani, and K. G. Vamvoudakis, “Distributed learning algorithm for non-linear differential graphical games,” Transactions of the Institute of Measurement and Control, vol. 39, no. 2, pp. 173–182, Jul. 2017.
[18] Q. Jiao, H. Modares, S. Xu, F.L. Lewis, and K.G. Vamvoudakis, “Multi-agent zero-sum differential graphical games for disturbance rejection in distributed control,” Automatica, vol. 69, pp. 24-34, 2016.
[19] F. Tatari, K.G. Vamvoudakis, and M. Mazouchi, “Optimal distributed learning for disturbance rejection in networked nonlinear games under unknown dynamics,” IET Control Theory and Applications, vol. 13, no. 17, pp.2838-2848, 2019.
[20] D. Liu, H. Li, and D. Wang, “Neural-network-based zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm,” Neurocomputing, vol. 110, pp. 92–100, 2013.
[21] W. Wang, X. Chen, and J. Du, “Model-free finite-horizon optimal control of discrete-time two-player zero-sum games,” International Journal of Systems Science, vol. 54, no. 1, pp. 167-179, 2023.
[22] R. Song and L. Zhu, “Stable value iteration for two-player zero-sum game of discrete-time nonlinear systems based on adaptive dynamic programming,” Neurocomputing, vol. 340, pp. 180–195, May 2019.
[23] M. Abouheaf, F. L. Lewis, K. G. Vamvoudakis, Sofie Haesaert, and R. Babuska, “Multi-agent discrete-time graphical games and reinforcement learning solutions,” Automatica, vol. 50, no. 12, pp. 3038–3053, 2014.
[24] M. I. Abouheaf, F. L. Lewis, M. S. Mahmoud, and D. G. Mikulski, “Discrete-time dynamic graphical games: model-free reinforcement learning solution,” Control Theory and Technology, vol. 13, no. 1, pp. 55–69, Feb. 2015.
[25] R. S. Sutton, and A. G. Barto, “Reinforcement Learning: An Introduction,” Robotica, vol. 17, no. 2, pp. 229-235, 1999.
[26] M. Taghian, A. Asadi, and R. Safabakhsh, “A reinforcement learning-based encoder-decoder framework for learning stock trading rules,” Journal of Artificial Intelligence and Data Mining, vol. 11, no. 1, pp. 103-118, 2023.
[27] B. Kiumarsi, H. Modares, and F. Lewis, “Reinforcement learning for distributed control and multi-player games,” In Handbook of Reinforcement Learning and Control, Springer, Cham: Springer International Publishing, pp. 7-27, 2021.
[28] H. Jiang, H. Zhang, G. Xiao, and X. Cui, “Data-based approximate optimal control for nonzero-sum games of multi-player systems using adaptive dynamic programming” Neurocomputing, vol. 275, pp. 192-199, 2018.
[29] H. Jiang, H. Zhang, Y. Luo, and X. Cui, “H-infinity control with constrained input for completely unknown nonlinear systems using data-driven reinforcement learning method,” Neurocomputing, vol. 237, pp. 226-234, 2017.
[30] P. Liu, H. Zhang, C. Liu, and H. Su, “Online dual-network-based adaptive dynamic programming for solving partially unknown multi-player non-zero-sum games with control constraints,” IEEE Access, vol. 8, pp. 182295–182306, 2020.
[31] B. Luo, D. Liu, and H. N. Wu, “Adaptive constrained optimal control design for data-based nonlinear discrete-time systems with critic-only structure,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 6, pp. 2099–2111, 2017.
[32] X. i, Y. Tang, and H. R. Karimi, “Consensus of multi-agent systems via fully distributed event-triggered
control,” Automatica, vol. 116, pp. 108898, 2020.
[33] Y. Y. Qian, L. Liu, and G. Feng, “Output consensus of heterogeneous linear multi-agent systems with adaptive event-triggered control,” IEEE Transactions on Automatic Control, vol. 64, no. 6, pp. 2606–2613, Jun. 2018.
[34] X. Li, Z. Sun, Y. Tang, and H. R. Karimi, “Adaptive event-triggered consensus of multiagent systems on directed graphs,” IEEE Transactions on Automatic Control, vol. 66, no. 4, pp. 1670–1685, 2020.
[35] S. Hu, D. Yue, X. Yin, X. Xie, and Y. Ma, “Adaptive event-triggered control for nonlinear discrete-time systems,” International Journal of Robust and Nonlinear Control, vol. 26, no. 18, pp. 4104–4125, 2016.
[36] L. Dong, X. Zhong, C. Sun, and H. He, “Adaptive event-triggered control based on heuristic dynamic programming for nonlinear discrete-time systems,” IEEE transactions on neural networks and learning systems, vol. 28, no. 7, pp. 1594–1605, Jul. 2017.
[37] Z. Wang, Q. Wei, D. Liu, and Y. Luo “Event-triggered adaptive control for discrete-time zero-sum games,” In 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1-7. IEEE, 2019.
[38] S. Khoo, L. Xie, and Z. Man, “Robust finite-time consensus tracking algorithm for multirobot systems,” IEEE/ASME transactions on mechatronics, vol. 14, no. 2, pp. 219–228, 2009.
[39] M. Abu-Khalaf and F. L. Lewis, “Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach,” Automatica, vol. 41, no. 5, pp. 779–791, May 2005.
)