LAGEA: Language Guided Embodied Agents for Robotic Manipulation

Robotic manipulation benefits from foundation models that describe goals, but today's agents still lack a principled way to learn from their own mistakes. We ask whether natural language can serve as feedback, an error-reasoning signal that helps embodied agents diagnose what went wrong and correct course. We introduce LAGEA (Language Guided Embodied Agents), a framework that turns episodic, schema-constrained reflections from a vision language model (VLM) into temporally grounded guidance for reinforcement learning. LAGEA summarizes each attempt in concise language, localizes the decisive moments in the trajectory, aligns feedback with visual state in a shared representation, and converts goal progress and feedback agreement into bounded, step-wise shaping rewardswhose influence is modulated by an adaptive, failure-aware coefficient. This design yields dense signals early when exploration needs direction and gracefully recedes as competence grows. On the Meta-World MT10 and Robotic Fetch embodied manipulation benchmarks, LAGEA improves average success over the state-of-the-art (SOTA) methods by 9.0% on random goals, 5.3% on fixed goals, and 17% on fetch tasks, while converging faster. These results support our hypothesis: language, when structured and grounded in time, is an effective mechanism for teaching robots to self-reflect on mistakes and make better choices. Code will be released soon.

Robots often learn manipulation tasks through trial and error, but the feedback they receive is usually sparse: the task either succeeds or fails. This makes it difficult for the robot to know what went wrong, when the mistake happened, and how to improve on the next attempt. We introduce LAGEA, a framework that uses natural language as feedback for robot learning. After a robot attempts a task, a vision language model looks at key moments from the episode and produces a short, structured reflection describing the outcome, the likely error, and a suggested correction. LAGEA then connects this feedback to the robot’s visual observations and converts it into step-wise learning signals. These signals guide the robot more strongly when it is still failing and gradually reduce their influence as the robot becomes more competent. We evaluate LAGEA on simulated robotic manipulation benchmarks, including Meta-World MT10 and Robotic Fetch. LAGEA learns faster and achieves higher success than prior reinforcement learning baselines. These results suggest that structured language feedback can help embodied agents learn from their mistakes more effectively in sparse-reward manipulation tasks.