Moving Out: Physically-grounded Human-AI Collaboration

The ability to adapt to physical actions and constraints in an environment is crucial for embodied agents (e.g., robots) to effectively collaborate with humans. Such physically grounded human-AI collaboration must account for the increased complexity of the continuous state-action space and constrained dynamics caused by physical constraints. However, most existing collaboration benchmarks are discrete or do not consider physical attributes and constraints. To address this, we introduce Moving Out, a human-AI collaboration benchmark that resembles a wide range of collaboration modes affected by physical attributes and constraints, such as moving heavy items together and coordinating actions to move an item around a corner. Moving Out consists of two challenges and human-human interaction data to comprehensively evaluate models' abilities to adapt to diverse human behaviors and unseen physical attributes. To give embodied agents the capability to collaborate with humans under physical attributes and constraints, we propose a novel method, BASS (Behavior Augmentation, Simulation, and Selection), to enhance the diversity of agents and their understanding of the outcome of actions. We systematically compare BASS and state-of-the-art models in AI-AI and human-AI experiments, showing that BASS can effectively collaborate with both unseen AI and humans.

This paper studies how AI agents can better collaborate with humans in physical environments, such as moving large furniture together or carrying objects through narrow spaces. Existing Human-AI collaboration benchmarks are usually too simple and do not capture the challenges of real physical interactions. To address this, we introduce Moving Out, a new benchmark with realistic physics, continuous actions, and diverse collaboration tasks. We also collected over 1,000 pairs of human-human demonstrations to capture different human collaboration behaviors. In addition, we propose BASS, a method that improves collaboration by exposing AI agents to more diverse behaviors and allowing them to predict the outcomes of actions before acting. Experiments show that BASS works better than existing methods in both AI-AI and human-AI collaboration. Human participants also rated BASS as more helpful and better at understanding physical interactions. Our work provides a new benchmark and method for building AI systems that can collaborate with humans more naturally in the physical world.