Seeing Symbols, Missing Structure: A Real-World Handwritten Mathematical Expression Recognition Benchmark for Large Models

Handwritten mathematical expression recognition (HMER) remains challenging in real-world educational scenarios, even with recent advances in large vision-language models. While these models often achieve high accuracy in local symbol transcription, their reliability in capturing two-dimensional mathematical structure under realistic handwritten conditions is still poorly understood. We introduce a real-world handwritten benchmark covering 13 categories of structurally complex expressions with authentic writing artifacts. Evaluations on large models reveal a clear performance degradation as structural complexity increases, even when symbol-level accuracy is high. Most failures arise from structural mis-parsing and context-dependent symbol role confusion rather than pure visual perception errors. To mitigate this issue, we propose a training-free, schema-anchored structure-aware inference framework that decomposes recognition into schema identification, schema-constrained transcription, and context-driven disambiguation. Our method improves the ExpRate from 11.63\% to 24.52\% on Qwen-8B and generalizes well across multiple large models. Our benchmark provides a realistic evaluation for large models on handwritten mathematics, and our framework offers an effective and interpretable solution to structure-related failures in real-world HMER. Code and data are available at: https://github.com/BNU-ERC-ITEA/HMER-Bench.git.

Many students, teachers, and education platforms need to turn handwritten math into digital formulas, but this remains difficult for today’s AI systems. Although large vision-language models can often recognize individual handwritten symbols, they frequently misunderstand how those symbols are arranged in a mathematical expression. For example, they may confuse a matrix with separate lines, miss the alignment in vertical arithmetic, or ignore the meaning of crossed-out terms in cancellation steps. We introduce HMER-Bench, a real-world benchmark designed to test whether AI systems can understand not only handwritten symbols, but also the structure of handwritten mathematics. The benchmark contains diverse examples from educational scenarios, including long formulas, matrices, derivations, vertical arithmetic, chemical equations, and expressions with handwriting artifacts. Our experiments show that the main weakness of current models is not simply seeing symbols, but understanding mathematical layout and structure. To address this, we propose a training-free method that first identifies the overall expression type, then uses this structure to guide recognition and resolve ambiguous symbols. This makes recognition more accurate, stable, and easier to interpret. Our benchmark and method can help future AI systems better support math education, automatic grading, and handwritten document understanding.