ExpAlign: Expectation-Guided Vision–Language Alignment for Open-Vocabulary Grounding

Open-vocabulary grounding requires accurate vision-language alignment under weak supervision, yet existing methods either rely on global sentence embeddings that lack fine-grained expressiveness or introduce token-level alignment with explicit supervision or heavy cross-attention designs. We propose \textbf{ExpAlign}, a theoretically grounded vision-language alignment framework built on a principled multiple instance learning formulation. ExpAlign introduces an Expectation Alignment Head that performs attention-based soft MIL pooling over token-region similarities, enabling implicit token and instance selection without additional annotations. To further stabilize alignment learning, we develop an energy-based multi-scale consistency regularization scheme, including a Top-K multi-positive contrastive objective and a Geometry-Aware Consistency Objective derived from a Lagrangian-constrained free-energy minimization. Extensive experiments show that ExpAlign consistently improves open-vocabulary detection and zero-shot instance segmentation, particularly on long-tail categories. Most notably, it achieves 36.2 AP$_r$ on the LVIS minival split, outperforming other state-of-the-art methods at comparable model scale, while remaining lightweight and inference-efficient.

Artificial intelligence models often struggle to accurately find and outline specific objects in images based on open-ended, everyday language descriptions. Existing methods either produce blurry, imprecise results by compressing entire sentences into a single concept, or they demand massive computing power and tedious, manual human annotations to map out every single word. To bridge this gap, researchers developed a lightweight framework called ExpAlign to optimize word-to-image matching. Without relying on expensive manual data labeling, ExpAlign uses a smart mathematical mechanism to automatically calculate how strongly different image regions relate to specific words. It also features a unique geometric consistency technique that ensures the model locks onto coherent object shapes rather than fragmented boundaries. Across multiple benchmarks, ExpAlign demonstrates an exceptional ability to spot and precisely outline rare or unfamiliar objects that it was never explicitly trained to recognize. It substantially outperforms dominant, heavier state-of-the-art vision models on long-tail categories. Crucially, the module delivers these high-precision maps while remaining incredibly fast, lightweight, and efficient for practical deployment.