Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark

Grading in large undergraduate STEM courses often yields minimal feedback due to heavy instructional workloads. We present a large-scale empirical study of AI grading on real, handwritten single-variable calculus work from the University of California, Irvine. Using OCR-conditioned large language models with structured, rubric-guided prompting, our system produces scores and formative feedback for thousands of handwritten, in-person proctored free-response quiz submissions from nearly 800 students included in the paper’s empirical analysis. In a setting with no single ground-truth label, we evaluate performance against official teaching-assistant grades, student surveys, and independent human review, finding strong alignment with TA scoring and a large majority of AI-generated feedback rated correct or acceptable across quizzes. Beyond calculus, this setting highlights core challenges in OCR-conditioned mathematical reasoning and partial-credit assessment. We analyze key failure modes, propose practical rubric- and prompt-design principles, and introduce a multi-perspective evaluation protocol for reliable real-course deployment. Building on the dataset and evaluation framework developed here, we outline a path toward a future standardized benchmark for AI grading of handwritten mathematics to support reproducible evaluation, transparent comparison, reliable deployment, and future research.

This paper presents a large-scale study of how AI can assist with grading handwritten mathematics work in university courses. In large calculus classes, teaching assistants often grade many free-response solutions under tight deadlines, leaving limited time for detailed feedback. We evaluated an AI-assisted grading system on thousands of real handwritten calculus quiz solutions from nearly 800 students at UC Irvine. The system reads handwritten work, applies grading rubrics, assigns scores, and generates feedback. An important part of the system design was developing AI prompts and grading rubrics to help the system interpret handwritten solutions more accurately and evaluate different valid solution methods more fairly and consistently. We compare AI scores with teaching-assistant scores, collect student survey responses, and use independent reviewers to assess handwriting recognition and AI-generated feedback. Overall, AI scores showed strong agreement with human grading, and most AI feedback was judged correct or acceptable. The study also identifies practical challenges, including handwriting-recognition errors, inconsistent partial-credit decisions, and deciding when human review is needed. We also outline plans for future benchmark materials to help researchers evaluate and improve AI grading systems.