Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting

While Vision-Language-Action (VLA) models generalize well to generic instructions, they struggle with personalized commands such as "bring my cup," where the robot must act on one specific instance among visually similar objects. We study this setting of manipulating personal objects, in which a VLA must identify and control a user-specific object unseen during training using only a few reference images. To address this challenge, we propose Visual Attentive Prompting (VAP), a simple-yet-effective training-free perceptual adapter that equips frozen VLAs with top-down selective attention. VAP treats the reference images as a non-parametric visual memory, grounds the personal object in the scene through open-vocabulary detection and embedding-based matching, and then injects this grounding as a visual prompt by highlighting the object and rewriting the instruction. We construct two simulation benchmarks, Personalized-SIMPLER and Personalized-VLABench, and a real-world tabletop benchmark to evaluate personalized manipulation across multiple robots and tasks. Experiments show that VAP consistently outperforms generic policies and token-learning baselines in both success rate and correct-object manipulation, helping to bridge the gap between semantic understanding and instance-level control.

Robots are becoming better at following general commands such as “pick up the cup,” but they still struggle with personal commands such as “bring my cup.” In everyday homes, a robot may need to choose one specific object from several similar ones, even though that object was never seen during training. Language alone is often not enough to describe the small visual details that make a personal item unique. We study this problem and introduce Visual Attentive Prompting, or VAP, a training-free way to help existing robot models handle personal objects. The user provides a few reference photos of the desired object, and VAP uses them as a visual memory to find the matching object in the robot’s camera view. It then highlights the object in the image and rewrites the instruction so the robot can focus on the correct target without retraining. Across simulation and real-world robot experiments, VAP improves both choosing the right object and completing the manipulation task. This suggests a practical path toward household robots that can better adapt to individual users and their personal belongings.