Timezone: »

 
Poster
Deconfounded Value Decomposition for Multi-Agent Reinforcement Learning
Jiahui Li · Kun Kuang · Baoxiang Wang · Furui Liu · Long Chen · Changjie Fan · Fei Wu · Jun Xiao

Tue Jul 19 03:30 PM -- 05:30 PM (PDT) @ Hall E #803
in Poster Session 1 »

Value decomposition (VD) methods have been widely used in cooperative multi-agent reinforcement learning (MARL), where credit assignment plays an important role in guiding the agents’ decentralized execution. In this paper, we investigate VD from a novel perspective of causal inference. We first show that the environment in existing VD methods is an unobserved confounder as the common cause factor of the global state and the joint value function, which leads to the confounding bias on learning credit assignment. We then present our approach, deconfounded value decomposition (DVD), which cuts off the backdoor confounding path from the global state to the joint value function. The cut is implemented by introducing the \textit{trajectory graph}, which depends only on the local trajectories, as a proxy confounder. DVD is general enough to be applied to various VD methods, and extensive experiments show that DVD can consistently achieve significant performance gains over different state-of-the-art VD methods on StarCraft II and MACO benchmarks.

Keywords: RL: Multi-agent

Author Information

Jiahui Li (Zhejiang University)
Kun Kuang (Zhejiang University)
Kun Kuang

Kun Kuang is an Associate Professor at the College of Computer Science and Technology, Zhejiang University. He received his Ph.D. in the Department of Computer Science and Technology at Tsinghua University in 2019. He was a visiting scholar with Prof. Susan Athey's Group at Stanford University. His main research interests include Causal Inference, Data Mining, and Causality Inspired Machine Learning. He has published over 70 papers in prestigious conferences and journals in data mining and machine learning, including TKDE, TPAMI, ICML, NeurIPS, KDD, ICDE, WWW, MM, DMKD, Engineering, etc. He received ACM SIGAI China Rising Star Award in 2022.

Baoxiang Wang (The Chinese University of Hong Kong, Shenzhen)
Furui Liu (Huawei Noah's Ark Lab)
Long Chen (Columbia University)
Changjie Fan (NetEase Fuxi AI Lab)
Fei Wu (Zhejiang University)
Jun Xiao (Zhejiang University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors