Plan in Sandbox, Navigate in Open Worlds: Learning Physics-Grounded Abstracted Experience for Embodied Navigation

Vision-Language Models (VLMs) have demonstrated exceptional general reasoning capabilities. However, their performance in embodied navigation remains hindered by a scarcity of aligned open-world vision and robot control data. Despite simulators providing a cost-effective alternative for data collection, the inherent reliance on photorealistic simulations often limits the transferability of learned policies. To this end, we propose **Sandbox-Abstracted Grounded Experience (SAGE), a framework that enables agents to learn within a physics-grounded semantic abstraction rather than a photorealistic simulation, mimicking the human capacity for mental simulation where plans are rehearsed in simplified physics abstractions before execution. SAGE operates via three synergistic phases: (1) Genesis: constructing diverse, physics-constrained semantic environments to bootstrap experience; (2) Evolution: distilling experiences through Reinforcement Learning (RL), utilizing a novel asymmetric adaptive clipping mechanism to stabilize updates; (3) Navigation: bridging the abstract policy to open-world control. We demonstrate that SAGE significantly improves planner-assisted embodied navigation, achieving a 53.21% LLM-Match Success Rate on A-EQA (+9.7% over baseline), while showing encouraging transfer to physical indoor robot deployment. Project page is available at: .

Imagine asking a household robot, "What's on the kitchen table?" To answer, the robot needs to combine language understanding with physical exploration. Today's AI is strong on the language side but struggles to link what it sees to where to move next, and the real-world examples needed to teach this link are scarce. Realistic 3D simulators help, but skills learned there often break down once a real robot has to deal with a messy room. We took inspiration from how people think before acting. Our system SAGE lets a robot rehearse in a similar stripped-down sandbox: a world that keeps the essential layouts, walls, and objects. The robot invents its own practice questions, takes virtual trips through this sandbox, and learns from them using a training method that absorbs good examples quickly without becoming unstable. It then carries those skills into real homes, deciding at each step whether to explore further or revisit a place it remembers. On standard navigation benchmarks, SAGE outperforms previous methods by nearly ten percent, and a small open-source version even beats much larger commercial systems. We also deployed it on a real indoor robot that successfully answered questions and found objects on command, showing that simplified mental rehearsal can be a practical way to teach robots to act in the real world.