Timezone: »
Value decomposition (VD) methods have been widely used in cooperative multi-agent reinforcement learning (MARL), where credit assignment plays an important role in guiding the agents’ decentralized execution. In this paper, we investigate VD from a novel perspective of causal inference. We first show that the environment in existing VD methods is an unobserved confounder as the common cause factor of the global state and the joint value function, which leads to the confounding bias on learning credit assignment. We then present our approach, deconfounded value decomposition (DVD), which cuts off the backdoor confounding path from the global state to the joint value function. The cut is implemented by introducing the \textit{trajectory graph}, which depends only on the local trajectories, as a proxy confounder. DVD is general enough to be applied to various VD methods, and extensive experiments show that DVD can consistently achieve significant performance gains over different state-of-the-art VD methods on StarCraft II and MACO benchmarks.
Author Information
Jiahui Li (Zhejiang University)
Kun Kuang (Zhejiang University)

Kun Kuang is an Associate Professor at the College of Computer Science and Technology, Zhejiang University. He received his Ph.D. in the Department of Computer Science and Technology at Tsinghua University in 2019. He was a visiting scholar with Prof. Susan Athey's Group at Stanford University. His main research interests include Causal Inference, Data Mining, and Causality Inspired Machine Learning. He has published over 70 papers in prestigious conferences and journals in data mining and machine learning, including TKDE, TPAMI, ICML, NeurIPS, KDD, ICDE, WWW, MM, DMKD, Engineering, etc. He received ACM SIGAI China Rising Star Award in 2022.
Baoxiang Wang (The Chinese University of Hong Kong, Shenzhen)
Furui Liu (Huawei Noah's Ark Lab)
Long Chen (Columbia University)
Changjie Fan (NetEase Fuxi AI Lab)
Fei Wu (Zhejiang University)
Jun Xiao (Zhejiang University)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Spotlight: Deconfounded Value Decomposition for Multi-Agent Reinforcement Learning »
Tue. Jul 19th 09:25 -- 09:30 PM Room Room 318 - 320
More from the Same Authors
-
2022 : Towards Multi-level Fairness and Robustness on Federated Learning »
Fengda Zhang · Kun Kuang · Yuxuan Liu · Long Chen · Jiaxun Lu · Yunfeng Shao · Fei Wu · Chao Wu · Jun Xiao -
2023 Poster: Causal Structure Learning for Latent Intervened Non-stationary Data »
Chenxi Liu · Kun Kuang -
2023 Poster: Stable Estimation of Heterogeneous Treatment Effects »
Anpeng Wu · Kun Kuang · Ruoxuan Xiong · Bo Li · Fei Wu -
2022 Poster: The Role of Deconfounding in Meta-learning »
Yinjie Jiang · Zhengyu Chen · Kun Kuang · Luotian Yuan · Xinhai Ye · Zhihua Wang · Fei Wu · Ying WEI -
2022 Poster: Instrumental Variable Regression with Confounder Balancing »
Anpeng Wu · Kun Kuang · Bo Li · Fei Wu -
2022 Poster: Individual Reward Assisted Multi-Agent Reinforcement Learning »
Li Wang · Yupeng Zhang · Yujing Hu · Weixun Wang · Chongjie Zhang · Yang Gao · Jianye Hao · Tangjie Lv · Changjie Fan -
2022 Spotlight: Instrumental Variable Regression with Confounder Balancing »
Anpeng Wu · Kun Kuang · Bo Li · Fei Wu -
2022 Spotlight: Individual Reward Assisted Multi-Agent Reinforcement Learning »
Li Wang · Yupeng Zhang · Yujing Hu · Weixun Wang · Chongjie Zhang · Yang Gao · Jianye Hao · Tangjie Lv · Changjie Fan -
2022 Spotlight: The Role of Deconfounding in Meta-learning »
Yinjie Jiang · Zhengyu Chen · Kun Kuang · Luotian Yuan · Xinhai Ye · Zhihua Wang · Fei Wu · Ying WEI -
2021 Poster: MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration »
Jin Zhang · Jianhao Wang · Hao Hu · Tong Chen · Yingfeng Chen · Changjie Fan · Chongjie Zhang -
2021 Spotlight: MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration »
Jin Zhang · Jianhao Wang · Hao Hu · Tong Chen · Yingfeng Chen · Changjie Fan · Chongjie Zhang -
2021 Poster: Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework »
Wenxiao Wang · Minghao Chen · Shuai Zhao · Long Chen · Jinming Hu · Haifeng Liu · Deng Cai · Xiaofei He · Wei Liu -
2021 Spotlight: Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework »
Wenxiao Wang · Minghao Chen · Shuai Zhao · Long Chen · Jinming Hu · Haifeng Liu · Deng Cai · Xiaofei He · Wei Liu -
2021 Poster: Explainable Automated Graph Representation Learning with Hyperparameter Importance »
Xin Wang · Shuyi Fan · Kun Kuang · Wenwu Zhu -
2021 Spotlight: Explainable Automated Graph Representation Learning with Hyperparameter Importance »
Xin Wang · Shuyi Fan · Kun Kuang · Wenwu Zhu -
2019 Poster: Disentangled Graph Convolutional Networks »
Jianxin Ma · Peng Cui · Kun Kuang · Xin Wang · Wenwu Zhu -
2019 Oral: Disentangled Graph Convolutional Networks »
Jianxin Ma · Peng Cui · Kun Kuang · Xin Wang · Wenwu Zhu