HiCI: Hierarchical Construction–Integration for Long-Context Attention

Long-context language modeling is commonly framed as a scalability challenge of token-level attention, yet local-to-global information structuring remains largely implicit in existing approaches. Drawing on cognitive theories of discourse comprehension, we propose HiCI (Hierarchical Construction--Integration), a hierarchical attention module that constructs segment-level representations, integrates them into a shared global context, and broadcasts both to condition segment-level attention. We validate HiCI through parameter-efficient adaptation of LLaMA-2 (7B and 13B) and Qwen3-8B with only $\sim$4--5\% additional parameters, extending context to 100K/64K tokens for LLaMA-2-7B/13B and to 48K tokens for Qwen3-8B. Across language modeling, retrieval, and instruction-following benchmarks, HiCI yields consistent improvements over strong baselines, including matching proprietary models on topic retrieval and surpassing GPT-3.5-Turbo-16K on code comprehension. These results demonstrate the effectiveness of explicit hierarchical structuring as an inductive bias for long-context modeling.

Modern AI language models—the technology behind chatbots like ChatGPT—work well on short text but struggle with very long documents. Reading lengthy legal contracts or research papers requires tracking details across many pages, which current models cannot reliably do without losing context. We introduce HiCI, inspired by how humans handle long-form material. As the model processes a document, HiCI summarizes each section into a few key points, organizes these into a global overview of the entire document, and uses this overview when interpreting later parts of the text. We evaluate HiCI on three open-source language models. With only a small amount of additional training, HiCI enables them to process inputs roughly the length of a novel, improves their accuracy in retrieving specific facts from long documents, and matches the performance of leading commercial systems such as GPT-4 and Claude on selected benchmarks. By improving reliability on long inputs, this work expands the practical applicability of language models to tasks such as legal review, scientific research, and extended conversations.