MapDream: Task-Driven Map Learning for Vision-Language Navigation

Vision-Language Navigation (VLN) requires agents to follow natural language instructions in partially observed 3D environments, motivating map representations that aggregate spatial context beyond local perception. However, most existing approaches rely on hand-crafted maps constructed independently of the navigation policy. We argue that maps should instead be learned representations shaped directly by navigation objectives rather than exhaustive reconstructions. Based on this insight, we propose MapDream, a map-in-the-loop framework that formulates map construction as autoregressive bird’s-eye-view (BEV) image synthesis. The framework jointly learns map generation and action prediction, distilling environmental context into a compact three-channel BEV map that preserves only navigation-critical affordances. Supervised pre-training bootstraps a reliable mapping-to-control interface, while the autoregressive design enables end-to-end joint optimization through reinforcement fine-tuning. Experiments on R2R-CE and RxR-CE achieve state-of-the-art monocular performance, validating task-driven generative map learning.

Robots and embodied AI agents are often asked to follow language instructions such as “go down the hallway and turn left at the sofa.” This is difficult because the agent can only see a small part of the 3D environment at each moment, so it needs some form of memory to understand where it has been and where it should go next. Many existing methods use manually designed maps to store spatial information, but these maps are usually built separately from the agent’s decision-making process. In this work, we argue that a useful map should not simply reconstruct the whole environment. Instead, it should learn to keep the information that matters most for navigation. We propose MapDream, a framework that lets an agent generate a compact bird’s-eye-view map while learning how to navigate. This map contains three types of navigation-related information and is directly optimized together with the agent’s actions. Experiments on two standard navigation benchmarks show that MapDream achieves state-of-the-art performance using only monocular visual input.