Zero-shot Active Mapping via Fused 360-BEV Representations and Vision–Language Models

Active mapping enables embodied agents to understand and interact in previously unseen environments. However, most methods struggle to achieve zero-shot generalization to large-scale scenes and lack support for language instructions. We propose a VLM-based active mapping method that achieves zero-shot mapping while facilitating language-driven human–agent interaction. First, we introduce a 360-BEV representation that integrates omnidirectional semantics with BEV-aligned geometric structure to enhance scene understanding. Second, we develop a candidate waypoint generation strategy that allows the VLM-driven agent to select informative 2D waypoints in image space and back-project them into executable metric actions in 3D space, enabling the VLM to plan in its strongest modality. Third, we design a VLM-based depth-first exploration agent that decomposes the scenes into explorable regions, selects informative waypoints within each region, and organizes them into a topological tree. The agent follows the depth-first exploration policy to achieve thorough coverage of large-scale scenes. Without task-specific training, our method outperforms the strongest baseline, improving coverage and AUC by approximately 13.25\% and 14.00\%, respectively, while enabling language-conditioned interaction.

Robots and drones need to explore and map unknown environments, such as buildings or disaster sites, before they can navigate or assist humans in them. Current mapping methods require extensive training on specific environments and fail when placed in unfamiliar settings. They also cannot understand or follow human instructions like "go update the map of the bedroom." We developed a method that allows a flying robot to explore and map large indoor spaces it has never seen before, without any prior training on those environments. Our approach works by combining a panoramic camera view with a bird's-eye-view map, giving the robot a comprehensive understanding of its surroundings. A large vision-language AI model then analyzes this combined view to decide where to go next, organizing the environment into a tree-like structure of connected regions and exploring them systematically. Because the AI model understands both images and language, users can simply tell the robot which area to revisit or update using everyday language. In experiments, our method mapped significantly more of each environment than the best existing approach, all without any task-specific training, and successfully followed human language commands to update specific areas of the map.