L-CUBE: Isolating Long-Context Capacity from Knowledge with Controllable Mutual Information Scaling

Evaluating long-context language models on natural language conflates architectural capacity to capture dependencies with semantic knowledge and vocabulary statistics. When models fail at long contexts, we cannot determine whether failures stem from fundamental architectural limitations or insufficient domain knowledge, preventing clean diagnosis of efficient architectures before expensive training on real data. We introduce L-CUBE (Long-Context Utilization Benchmark), a synthetic benchmark that isolates dependency-capturing capacity from semantic knowledge through hierarchical Gaussian sequences with controllable bipartite mutual information scaling. The generator provides exact ground-truth conditionals that scale efficiently to arbitrarily long sequences, enabling unconfounded evaluation via conditional KL divergence rather than perplexity alone. We define long-context utilization to measure the amount of available predictive information that models extract as context grows. Experiments across transformers, state space models, and efficient alternatives validate L²M capacity theory predictions and uncover new phenomena. L-CUBE enables practitioners to test whether a particular design will maintain long-context capability at target sequence lengths before committing to real-data training.

When AI language models struggle with long documents, researchers face a difficult question: is the model's architecture fundamentally unable to handle long texts, or does it simply lack enough knowledge about the topic? Current tests cannot distinguish between these two causes, making it hard to know which AI designs will work for long documents without expensive trial and error. We created L-CUBE, a testing tool that separates these two issues. Instead of using real language, L-CUBE generates synthetic number sequences with precisely controlled long-range patterns, mimicking how a word in one paragraph can relate to a word many paragraphs later. Because we design these patterns ourselves, we know exactly how much long-range connection exists, giving us a clear answer key to grade AI architectures against. Using L-CUBE, we found that standard transformer models successfully capture long-range patterns across all tested lengths, while more computationally efficient alternatives like state space models eventually hit a ceiling and miss important connections. Interestingly, when accounting for computational cost, all architectures improve at similar rates, though they start from different baselines. L-CUBE lets developers quickly test whether a new design can handle long documents before investing months of training on real data.