MGAL: A Multilingual Granularity-Aware Long-Context Benchmark

Evaluation of long-context Large Language Models (LLMs) has advanced rapidly. However, most existing benchmarks are limited to the document level and focus mainly on high-resource languages, leaving many fine-grained challenges insufficiently evaluated. To address this gap, we present MGAL, the first multilingual, granularity- and position-aware long-context benchmark. MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages. It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document (begin, middle, and end), indexed at both the document and paragraph levels. This design enables systematic diagnosis of multilingual long-context comprehension across different granularities. Through extensive experiments and analyses, we find that: (1) LLMs perform well at word-level tasks but struggle with coarser-grained ones; and (2) Closed-source models retain a clear performance advantage in lower-resource languages. We further identify two new challenges: (1) Under local semantic crowding, where neighboring sentences share topics and entities, models tend to follow surface cues (e.g., connectives like 'however' or repeated entities) rather than the discourse role of the sentence in surrounding context (e.g., background, outcome); and (2) A gap between fluency and consistency in generated outputs, where models produce text that reads smoothly but drifts from the source facts. In addition, we observe several patterns in line with prior studies, including reliance on nearby evidence and reuse of options under uncertainty.

Modern Large Language Models can now read very long documents, but it is still unclear how well they truly understand them, especially across different languages and different levels of detail. Our work introduces MGAL, a new benchmark designed to test this ability more carefully. MGAL is built from long United Nations reports in the six official UN languages: Arabic, Chinese, English, French, Russian, and Spanish. These reports can be very long, ranging from thousands to more than one hundred thousand words or tokens. Unlike many existing tests that only ask models to understand a whole document, MGAL evaluates understanding at multiple levels: individual words, sentences, paragraphs, and full documents. It also checks whether models perform differently when important information appears near the beginning, middle, or end of a long text. This makes it possible to diagnose not only whether a model gives the right answer, but also where and why it struggles. We evaluate twelve strong language models and find that they are usually good at locating short, specific answers, but they become much weaker when they need to understand or generate longer pieces of text. We also find that models often rely on surface clues, such as repeated words or familiar connectors, instead of truly understanding the role of a sentence in context. In longer generation tasks, models may produce fluent and natural-sounding text that is not fully faithful to the original document. Overall, MGAL provides a more detailed and multilingual way to evaluate long-document understanding, helping researchers build models that are more reliable, fair, and faithful across languages and document structures.