Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning

Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks. Despite its flexibility, multimodal ICL incurs high inference latency and suffers from instability due to sensitivity to demonstration formatting, ordering, and content. To address these limitations, we propose Hyper-ICL, a lightweight, training-based framework for demonstration-free multimodal ICL that reconstructs demonstration effects directly without requiring ICDs at inference time. Hyper-ICL learns a parameter-efficient low-rank logit-level adapter that calibrates attention distributions to better match demonstration-induced attention redistribution. To capture how demonstration influence varies across queries, we introduce a query-adaptive modulation mechanism that adaptively controls intervention strength at token level across layers and heads based on the current query. Finally, we propose a layer-wise hyperbolic anchor distillation loss that aligns intermediate student features to a demonstration-conditioned teacher via Lorentz geodesic distance. This loss encourages the student to reconstruct the demonstration–query relationships induced by ICDs. Extensive experiments across six different multimodal benchmarks (including VQAv2, OK-VQA, and COCO Caption) demonstrate that Hyper-ICL consistently improves accuracy and stability over vanilla ICL and existing state-of-the-art methods.

Multimodal AI systems can answer questions about images, describe visual scenes, and reason over image-text inputs. A common way to improve these systems is to give them a few examples in the prompt, but these examples make the input much longer, slow down inference, and can make the model sensitive to the order or wording of the examples. We introduce Hyper-ICL, a method that teaches the model to capture the benefit of these examples without needing to include them at inference time. Instead of storing or repeating demonstrations, Hyper-ICL learns a small adapter that adjusts the model’s attention to focus on the kinds of visual and textual information that demonstrations would have highlighted. Hyper-ICL also uses layer-wise hyperbolic anchor distillation to help the model preserve useful relationships learned from demonstrations. Our results show that Hyper-ICL improves accuracy and stability across several vision-language benchmarks while keeping inference nearly as efficient as zero-shot prediction. This can make multimodal AI faster, more practical, and more energy-efficient for real-world applications.