DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents

Multimodal Large Language Models (MLLMs) have achieved strong performance on structured visual understanding tasks such as chart and document question answering. However, existing benchmarks typically evaluate these domains in isolation, leaving underexplored a key capability: whether models can use textual context to determine how chart evidence should be selected, interpreted, and aggregated. We introduce DocHop, a benchmark for integrated chart--context reasoning in document-style images. In DocHop, the document narrative specifies multi-step compositional constraints, while charts provide the corresponding data values. Questions are grounded on a semantic reference label defined in the narrative, requiring models to resolve target entities from context before aggregating evidence across multiple charts. To enable systematic evaluation, we construct DocHop via a stochastic logic-first generation pipeline with controllable reasoning depth and visual density, covering 1,876 examples across six task categories. Experiments on a wide range of proprietary and open-source MLLMs show a substantial gap to human performance: annotators achieve over 90% accuracy, while the best model reaches only 60.18%. Reasoning-enhanced models consistently show improved results, but performance degrades as reasoning complexity increases. Overall, DocHop provides a controlled testbed for challenging multi-hop document reasoning.

Many documents combine text and charts. For example, a report may explain which companies, hospitals, or regions meet certain conditions, while the actual numbers needed to answer a question are shown in several charts. Today’s AI systems are often good at reading text or understanding charts separately, but it is still unclear whether they can reliably connect the two. In this work, we introduce DocHop, a new test for evaluating this ability. Each DocHop example contains a document-style page with explanatory text and multiple charts. The text describes a series of conditions that identify a group of relevant items, and the charts contain the numerical evidence needed to answer a question about them. To answer correctly, an AI system must first understand the text, figure out which items are being referred to, and then find and combine the right information from the charts. We evaluate a wide range of modern AI models on DocHop and find that the task remains challenging. Human annotators achieve over 90% accuracy, while the best-performing model reaches about 60%. Models with stronger reasoning abilities perform better, but their performance still drops as the documents require more reasoning steps or contain more charts. These results suggest that connecting written context with visual evidence remains an important challenge for current multimodal AI systems.