SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes

Simulation has become a key tool for training and evaluating home robots at scale, yet existing environments fail to capture the diversity and physical complexity of real indoor spaces. Current scene synthesis methods produce sparsely furnished rooms that lack the dense clutter, articulated furniture, and physical properties essential for robotic manipulation. We introduce SceneSmith, a hierarchical agentic framework that generates simulation-ready indoor environments from natural language prompts. SceneSmith constructs scenes through successive stages—from architectural layout to furniture placement to small object population—each implemented as an interaction among VLM agents: designer, critic, and orchestrator. The framework tightly integrates asset generation through text-to-3D synthesis for static objects, dataset retrieval for articulated objects, and physical property estimation. SceneSmith generates 3-6x more objects than prior methods, with $<$2\% inter-object collisions and 96\% of objects remaining stable under physics simulation. In a user study with 205 participants, it achieves 92\% average realism and 91\% average prompt faithfulness win rates against baselines. We further demonstrate that these environments can be used in an end-to-end pipeline for automatic robot policy evaluation.

Developing useful home robots requires exposing them to many realistic homes before deployment, but collecting real robot data is slow, expensive, and risky. Simulation can help, yet current simulated homes are often too empty and simple: they lack the clutter, movable cabinets, small objects, and physical details that make real homes hard for robots. We built SceneSmith, a system that turns a plain-language description into a complete indoor environment that a robot simulator can use directly. It plans the layout, adds furniture, fills shelves and tables with individual objects, creates or retrieves 3D models, and adds physical properties and collision shapes needed for simulation. This makes scenes interactive rather than just visually plausible: a robot can touch objects, pick them up, open cabinets, and be evaluated on tasks. In a user study, room-level SceneSmith scenes were preferred in 92% of realism judgments and 91% of prompt-faithfulness judgments on average against prior methods. SceneSmith also produced 3-6 times more objects while keeping collisions low and most objects physically stable. We further show that these scenes can be used to automatically evaluate robot behaviors from written task instructions. This could make robot training and evaluation more scalable, diverse, and realistic.