Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation

In embodied vision-language decision making tasks such as robotic manipulation and navigation, Vision-Language and Vision-Language-Action Models (VLMs \& VLAs) are powerful tools with different benefits: VLMs are better at long-term planning, while VLAs are better at reactive control. However, their performance is limited by the same perceptual bottleneck: visual hallucinations arise due to the models’ inability to distinguish task-relevant objects from distractors. In principle, accurate identification and focus on critical objects while filtering out irrelevant ones is the key to break this limitation. A straightforward solution is one-step focus: directly attending to essential objects. However, this approach proves ineffective because effective focus inherently requires deep scene understanding. To this end, we propose ${\it SceneDiver}$, a coarse-to-fine focus plan generation method for VLMs leveraging their long-term planning abilities, that first constructs a holistic scene graph to establish initial comprehension, then progressively decomposes the task into simpler sub-problems through an iterative cycle of recognition, understanding, and analysis. To enable reactive control, we also design a lightweight adapter for distilling the deliberate focus ability into VLAs. Evaluations on standard embodied AI benchmarks confirm that our method substantially reduces visual hallucinations for both VLMs and VLAs, while preserving computational efficiency in tasks requiring fast execution. Our code and data are released at: https://future-item.github.io/SceneDiver.

In embodied vision-language decision making tasks such as robotic manipulation and navigation, Vision-Language Models and Vision-Language-Action Models are powerful tools with different benefits: Vision-Language Models are better at long-term planning, while Vision-Language-Action Models are better at reactive control. However, their performance is limited by the same perceptual bottleneck: the models cannot reliably distinguish task-relevant objects from distractors. In principle, accurate identification and focus on critical objects while filtering out irrelevant ones is the key to breaking this limitation. A straightforward solution is one-step focus, which directly attends to essential objects, but this proves ineffective because effective focus requires deep scene understanding. To address this, we propose SceneDiver, a coarse-to-fine focus plan generation method that first constructs a holistic scene graph to establish initial comprehension, then progressively decomposes the task into simpler sub-problems through recognition, understanding, and analysis. To enable reactive control, we also design a lightweight adapter that distills the deliberate focus ability into Vision-Language-Action Models. Experiments on embodied AI benchmarks show that SceneDiver helps both Vision-Language Models and Vision-Language-Action Models break the perceptual bottleneck, while preserving computational efficiency in tasks requiring fast execution.