MedDocBench: Benchmarking Vision-Language Models for Reliable Structured Extraction from Real-World Clinical Forms

Structured health data often begins as paper or scanned clinical forms before entering downstream analysis, yet the reliability of this ingestion step is rarely benchmarked with the same rigor as the modeling that follows. Clinical forms pose unique extraction challenges like noisy handwriting, domain-specific abbreviations, ambiguous checkbox marks, and scarce annotated resources that existing OCR and medical document benchmarks do not jointly address. We introduce MedDocBench, a bilingual benchmark covering two tasks: free-form transcription measured by character error rate (CER), normalized edit distance (NED), and bag-of-words F1 (BoW F1), and structured JSON extraction of handwritten text and yes/no checkboxes measured by handwriting CER and checkbox accuracy. A central contribution is a normalization-aware evaluator with an explicit ON/OFF postprocessing switch for bilingual key mapping, value normalization, and fuzzy matching. Across 12 VLMs, toggling this switch shifts weighted scores by 0.13--0.46 across models; transcription performance does not reliably predict structured extraction quality. These results show that evaluator design can substantially change measured model quality, affecting downstream cohort definitions and risk models.