CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement

While LLM-based agents excel at individual tasks, effective collaboration with realistic human partners remains challenging. Most of the existing conversation-level collaborative studies lack grounded interaction and behavioral execution, motivating the need for cooperative game environments that enable contextualized and immersive collaboration. To this end, this paper proposes CollabBench, a benchmark for evaluating and training collaborative agents in cooperative games. CollabBench features a Diverse Player Profile Simulation pipeline to model varied players behaviors, and a Collaborative Agentic Training paradigm that unifies reasoning, communication, and action via agentic rollouts, optimized with a hybrid reward balancing task efficiency and affective adaptation. We further extend classic environments to CWAH-MultiPlayer and Cook-MultiPlayer for systematic evaluation under diverse personalities. Experiments with efficiency and affective metrics show that our trained models outperform base models, achieving 19.5% higher efficiency and 24.4% improved affective performance. Further analysis reveals key collaborative limitations of existing models and offers insights for future collaborative training.

Many AI Agents perform well on individual tasks, but still struggle to collaborate effectively with human partners in realistic settings. In real teamwork such as gameplay, people adapt their behavior and rely on communication, while most existing studies only test AI Agents in simplified dialogue environments that miss these dynamics. To address this, we propose CollabBench, a benchmark for studying and training AI agents in cooperative game environments with more realistic interaction. We build a simulation pipeline that generates diverse player behaviors and personalities, enabling agents to interact with more human-like partners. We also design a training framework that integrates reasoning, communication, and action, and optimizes both task efficiency and interaction quality. We further extend classic cooperative games into multi-player versions with different personality settings for more comprehensive evaluation. Experiments show that our trained agents achieve higher efficiency and better social interaction quality than baseline models. These results highlight that effective AI collaboration requires not only task-solving ability, but also understanding and adapting to human behavior in shared environments.