Looking Locally: Object-Centric Vision Transformers as Foundation Models for Efficient Segmentation

Current state-of-the-art segmentation models encode entire images before focusing on specific objects. This wastes computational resources. We introduce FLIP (Fovea-Like Input Patching), a parameter-efficient vision model that realizes object segmentation through biologically-inspired top-down attention. FLIP selectively samples multi-resolution patches centered on objects of interest from the input. As a result, it allocates high-resolution processing to object centers while maintaining coarser peripheral context. This off-grid, scale-invariant design enables FLIP to outperform META's Segment Anything models (SAM, SAM2 and fast variants) by large margins: With more than 440 $ \times$ fewer parameters, FLIP-Tiny (0.51M parameters) reaches a mean IoU of 79.90\% while SAM2-L reaches 75.87\% IoU (224.45M parameters). FLIP-Large even achieves 83.26\% mean IoU (96.6M parameters), still running about $2 \times$ faster than SAM2-L. We evaluate on six benchmarks in total. In five established benchmarks (Hypersim, KITTI-360, OpenImages, COCO, LVIS) FLIP consistently outperforms SAM and various variants of it. In our novel ObjaScale dataset, which stress-tests scale invariance with objects ranging from 0.0001\% up to 25\% of the image area, we show that FLIP segments even very small objects accurately, where existing models fail severely. FLIP opens new possibilities for real-time, object-centric vision and offers much higher energy efficiency. We believe that FLIP can act as a powerful foundation model, as it is very well-suited to track objects over time, for example, when being integrated into slot-based scene segmentation architectures.

Many image-analysis systems first inspect every part of a picture before drawing the exact outline of a chosen object, which can consume unnecessary computational resources when the object is small or the image is very large. This paper introduces Fovea-Like Input Patching (FLIP), a model inspired by human vision: it looks in high detail at the target object and uses lower detail for the surrounding context. Instead of processing a whole image at one fixed level of detail, it samples small image pieces around an object the user points to and predicts which pixels belong to that object. The authors test the method on six image collections, including street scenes, indoor scenes, natural images, and a new stress test with objects ranging from tiny specks to large foreground items. Across these tests, it matches or improves on Segment Anything models while being much smaller, which helps it run faster and reduces hardware demands, with especially strong results for very small objects. This could make accurate object boundary computation more practical for real-time vision systems, though the current method still depends on a reliable cue that tells it where to look.