Revealing Long-context Potential of Attention Heads via Frequency Kernels

Large language model (LLM) exists a subset of attention heads that are highly responsible for long-context processing. Existing work has identified different long-context heads in models, but their detection methods mainly rely on model inference on actual long texts and do not analyze the inherent properties of the head parameters. In this paper, we use kernel methods to analyze static \textit{frequency kernels} formed by different rotation frequency components of attention heads, and we design a Long-context Potential Score (LPS) to measure the potential of attention heads in processing long contexts. Kernels of heads with high LPS exhibit concentrated low-frequency energy and low effective rank, which allow them to effectively capture highly specialized information from distant contexts. Experiments and analysis on long-context tasks and model behaviors show that the LPS metrics can well reflect the actual capability of heads on long contexts. Furthermore, by simply amplifying low-frequency kernels of heads with high retrieval potential, we can further improve model's performance on long-context tasks. Our metrics and head enhancement methods are fully static and offline, and they can be quickly conducted under low-resource constraints. Code is publicly available at here.

Large language models (LLMs) rely on attention heads to process information across long texts, but only a small subset of these heads appears to be especially important for handling long-context information. Previous studies identified such heads mainly by observing how models behave during inference on long documents, which can be computationally expensive and difficult to analyze. In this work, we introduce a new static analysis method based on kernel techniques to study the internal properties of attention heads directly from their parameters, without running long-context inference. We construct frequency kernels from the rotational frequency components of attention heads and propose a metric called the Long-context Potential Score (LPS) to estimate their ability to process long-range information. We find that heads with high LPS share clear structural properties: their kernels concentrate energy in low-frequency components and have low effective rank, allowing them to selectively retrieve important information from distant contexts. Experiments on long-context benchmarks show that LPS strongly correlates with the actual long-context capability of attention heads. In addition, we show that simply amplifying the low-frequency components of high-potential heads can further improve long-context performance. Our analysis and enhancement methods are fully static, offline, and computationally lightweight, making them practical even in low-resource environments.