FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization

In-context localization (ICL) seeks to localize a target object specified by a small set of support examples in a query image, operating on the fly without training or parameter updates. Despite rapid advances in vision–language models (VLMs), achieving category-agnostic and visually grounded ICL remains an open problem, even though it is essential for applications such as image editing, personalized visual search, and retrieval. Existing methods are fragile and rely on explicit category supervision, which not only limits applicability in realistic settings with unnamed or instance-specific objects but also introduces category bias that steers predictions toward semantic priors rather than visual evidence. We introduce a two-stage training framework that explicitly optimizes in-context attention between support bounding boxes and query images without category supervision. We further refine localization via reinforcement learning using Group Relative Policy Optimization (GRPO) to directly minimize localization error. This formulation enforces visual correspondence over semantic priors, yielding robust instance-level localization. Empirically, a 7B-parameter model trained with our objectives outperforms models up to 72B parameters, demonstrating that context-aware localization objectives can surpass scaling alone. Comprehensive ablations validate the contribution of each component.

People often want an AI system to find a specific object in a new image by showing it only a few examples, such as the same product, tool, or personal item in other photos. This is difficult because current vision-language models often depend on object names or broad categories, so they may look for “a chair” or “a shoe” instead of the exact item shown in the examples. Our work studies this problem without giving the model category labels or updating its parameters at test time. We train the model to pay direct visual attention to the marked objects in the support examples and then match them to the correct region in the query image. A second reinforcement-learning stage further improves the predicted box by rewarding accurate localization. The result is a smaller 7B model that can outperform much larger models, showing that better grounding and training objectives matter more than scale alone.