Poster

On Path to Multimodal Historical Reasoning: HistBench and HistAgent

Jiahao Qiu ⋅ Fulian Xiao ⋅ Yimin Wang ⋅ Yuchen Mao ⋅ Yijia Chen ⋅ Xinzhe Juan ⋅ Siran Wang ⋅ Xuan Qi ⋅ Tongcheng Zhang ⋅ Zixin Yao ⋅ Jiacheng Guo ⋅ Yifu Lu ⋅ Charles Argon ⋅ Jundi Cui ⋅ Daixin Chen ⋅ Junran Zhou ⋅ Shuyao Zhou ⋅ Zhanpeng Zhou ⋅ Ling Yang ⋅ Shilong Liu ⋅ Hongru WANG ⋅ Kaixuan Huang ⋅ xun jiang ⋅ Xi Gao ⋅ Mengdi Wang

Abstract

Recent advances in large language models (LLMs) have led to remarkable progress across various domains, yet their capabilities in the humanities, particularly history, remain underexplored. Historical reasoning poses unique challenges for LLMs, involving multimodal source interpretation, temporal inference, and cross-linguistic analysis. Existing general-purpose agents perform well on many current benchmarks but lack the domain expertise needed to address complex historical questions. To address this gap, we introduce HistBench, a new benchmark of 414 high-quality and carefully-reviewed questions stratified by difficulty and designed to evaluate LLM's capacity for historical reasoning. The tasks span a wide range of historical problems—from factual retrieval based on primary sources to interpretive analysis of manuscripts and images, to interdisciplinary challenges involving archaeology, linguistics, or cultural history. Furthermore, the benchmark dataset spans 29 ancient and modern languages and covers a wide range of historical periods and world regions. Finding the poor performance of LLMs and other agents on HistBench, we further present HistAgent, a history-specific agent equipped with carefully designed tools for OCR, translation, archival search, and image understanding in History. On HistBench, HistAgent based on GPT-4o achieves an accuracy of 27.54% pass@1 and 36.47% pass@2, significantly outperforming LLMs with online search and generalist agents, including GPT-4o (18.60%), DeepSeek-R1(14.49%), Grok 3(17.63%) and Open Deep Research by smolagents(20.29% pass@1 and 25.12% pass@2). These results highlight the limitations of existing LLMs and generalist agents and demonstrate the advantages of HistAgent for historical reasoning. Notably, HistAgent also achieves 60.00% pass@1 accuracy on the GAIA benchmark, showing that domain-specific customization doesn't hinder HistAgent's competitive performance on real-world general tasks.