See, Act, Adapt: Active Perception for Unsupervised Cross-Domain Visual Adaptation via Personalized VLM-Guided Agent

Pre-trained perception models excel in generic image domains but degrade significantly in novel environments like indoor scenes. The conventional remedy is fine-tuning on downstream data which incurs catastrophic forgetting of prior knowledge and demands costly, scene-specific annotations. We propose a paradigm shift through Sea$^2$ ($\textbf{Se}$e, $\textbf{A}$ct, $\textbf{A}$dapt): rather than adapting the perception modules themselves, we adapt how they are deployed through an intelligent pose-control agent. Sea$^2$ keeps all perception modules frozen, requiring no downstream labels during training, and uses only scalar perceptual feedback to navigate the agent toward informative viewpoints. Specially, we transform a vision-language model (VLM) into a low-level pose controller through a two-stage training pipeline: first fine-tuning it on rule-based exploration trajectories that systematically probe indoor scenes, and then refining the policy via unsupervised reinforcement learning that constructs rewards from the perception module’s outputs and confidence. Unlike prior active perception methods that couple exploration with specific models or collect data for retraining them, Sea$^2$ directly leverages off-the-shelf perception models for various tasks without the need for retraining. We conducted experiments on three visual perception tasks, including visual grounding, segmentation and 3D box estimation, with performance improvements of 13.54\%, 15.92\% and 27.68\% respectively on dataset ReplicaCAD.

Many AI systems that understand images work well on common internet photos, but often perform much worse in new environments such as indoor 3D scenes. A common fix is to retrain the model on new data, but this is expensive, requires labels, and can make the model forget what it previously learned. Our work explores a different idea: instead of changing the vision model itself, we teach an embodied agent to move to better viewpoints. The agent learns how to look around a scene so that frozen off-the-shelf perception models can make better predictions. It does this without new human annotations and without updating the perception models’ parameters. We test this idea on three tasks: locating an object from a text description, segmenting an object, and estimating its 3D box. Across these tasks, the learned agent consistently improves results by finding more informative views. This matters because it suggests that, in new environments, better perception is not only about building better models, but also about gathering better observations. The approach offers a practical way to improve visual systems while reducing retraining costs and avoiding loss of prior knowledge.