ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark

Large language models (LLMs) are increasingly applied to symbolic mathematics, yet existing evaluations often conflate pattern memorization with genuine reasoning. To address this gap, we present ASyMOB, a high-resolution dataset of 35,368 validated symbolic math problems spanning integration, limits, differential equations, series, and hypergeometrics. Unlike prior benchmarks, ASyMOB systematically perturbs each seed problem using symbolic, numeric, and equivalence-preserving transformations, enabling a fine-grained assessment of generalization and robustness. Our evaluation reveals three key findings: (1) most models’ performance collapses under minor perturbations, while top systems exhibit an apparent regime shift in robustness; (2) integrated code tools stabilize performance, particularly for weaker models; and (3) we identify examples where Computer Algebra Systems (CAS) fail while LLMs succeed, as well as problems solved only via a hybrid LLM-CAS approach, highlighting a promising integration frontier. ASyMOB serves as a principled diagnostic tool for measuring and accelerating progress toward building verifiable, trustworthy AI for scientific discovery.

Today’s large language models can often give convincing answers to math problems, but it is hard to tell whether they truly understand the math or are mostly recognizing familiar patterns. We built ASyMOB: starting with a test set of symbolic math problems, such as integrals, limits, and differential equations, we systematically complicated each problem while preserving its underlying mathematical idea. This method provides “dials” that introduce different types of complexity to the problems in a controlled manner, while also making the problems less familiar to the AI - so it can't simply "remember" the answer. We found that many models fail under even small changes, while the strongest models are surprisingly robust. We also found cases where computer math programs fail but AI models succeed, and cases that were solved only by combining the flexibility of AI with the rigor of traditional tools. Scientists and engineers need AI systems that remain reliable when problems appear in new forms, not only when they resemble examples seen before. By exposing where current systems break and where tool use helps, ASyMOB can guide future AI systems toward more trustworthy and verifiable support for mathematical and scientific work.