Reasoning or Recall? Blind Evaluation of LLMs on Numerical Function Tasks
Abstract
For numerical tasks, existing LLM evaluations typically measure either the model's ability to generate code or to recall information. We examine a complementary capability: direct numerical inference from data, without tool use. To isolate it, we apply a uniform protocol across three tasks: multifidelity surrogate modeling, numerical integration, and global optimization. The input for each task is a table of point evaluations from a function whose formula is unknown --- a setting typical of uncertainty quantification problems. To focus on reasoning, rather than recall or code execution, we ask LLMs not to guess the underlying function or run code. We evaluated 4 LLMs and standard test functions across growing sample sizes. Models were prompted to select a method (from a list) for each task, then predict the function at unseen query points, estimate the integral, and locate the global minimum (each within a per-task tolerance). LLMs often converged to the same method for the same task, regardless of the training points, while prediction quality and confidence clearly scaled up with sample size. A fraction of answers matched the ground truth to several decimal places, well beyond what sparse-sample interpolation can yield, a pattern consistent with recall rather than reasoning. We probed this with a control function we composed, where recall is not possible: no more matches occurred.