PRISM: Perception Reasoning Interleaved for Sequential Decision Making.

Scaling LLM-based embodied agents from text-only environments to complex multimodal settings remains a major challenge. Recent work identifies a perception–reasoning–decision gap in standalone Vision–Language Models (VLMs), which often overlook task-critical information. In this paper, we introduce PRISM, a framework that tightly couples perception (VLM) and decision (LLM) through a dynamic question–answer (DQA) pipeline. Instead of passively accepting the VLM’s description, the LLM critiques it, probes the VLM with goal-oriented questions, and synthesizes a compact image description. This closed-loop interaction yields a sharp, task-driven understanding of the scene. We evaluate PRISM on the ALFWorld and Room-to-Room (R2R) benchmarks. We show that: (1) PRISM significantly outperforms state-of-the-art image-based models, (2) our Interactive goal-oriented perception pipeline yields systematic and substantial gains, and (3) PRISM is fully automatic, eliminating the need for handcrafted questions or answers.

Problem: Teaching autonomous agents to navigate and autonomously interact with the physical world is a major challenge. Often, the agent's "eyes" (vision models) and "brain" (reasoning models) operate too independently, causing the system to overlook crucial visual details needed to achieve a specific goal. Solution: To fix this, we developed PRISM, a framework that tightly couples the agent's vision and decision-making systems. Instead of the vision system passively providing a generic, overarching summary of a room, the reasoning system critiques the description and asks specific, goal-driven questions, such as, "Do you see a drawer nearby?". The system then synthesizes these targeted answers into a sharp, highly relevant description of its surroundings. Impact: This continuous closed-loop interaction gives the agent a much more accurate, task-driven understanding of its environment. As a result, PRISM significantly outperforms existing models in completing complex virtual household tasks and navigating realistic rooms. Ultimately, this brings us closer to building highly autonomous systems and robots that can reliably bridge the gap between decision-making and physical action in the real world.