SciKernelBench: Evaluating LLM Kernels for Higher-Order Autodiff and Scientific Residuals

Existing kernel generation benchmarks mainly target standard tensor operations such as matrix multiplications, convolutions, and elementwise operators. In contrast, scientific residuals require gradients, Jacobians, Laplacians, divergence, normal derivatives, and convective terms, often involving higher order autodiff paths that are not generally supported by current PyTorch compiler pipelines. We introduce \textbf{SciKernelBench}, a benchmark of 144 TorchPhysics derived workloads represented in the KernelBench format. The benchmark covers both isolated differential operator primitives and composed residual computations from physics informed neural networks, Deep Ritz formulations, inverse problems, time-dependent PDEs, contact mechanics, interface problems, and flow systems. SciKernelBench evaluates generated implementations by numerical correctness, runtime performance, and implementation honesty. The last axis is essential because candidates may pass ordinary output checks by reusing PyTorch autograd, preserving the reference path, omitting derivative terms, or hard coding deterministic inputs, without producing genuine low level acceleration. Across recent frontier LLMs and test time strategies, we find that scientific residuals remain challenging. On Level 1, Claude solves all tasks under repeated sampling and refinement without execution feedback, but drops to (23/84) raw and only (3/84) honest solves under refinement with execution feedback. Gemini shows a large raw honest gap even without feedback, with (33/84) raw but only (11/84) honest solves. On Level 2, Claude reaches (60/60) raw solves under execution feedback, while Gemini and Qwen remain substantially lower. Standard correctness based success can substantially overestimate genuine kernel generation ability in several settings, especially under refinement with execution feedback. These results show that scientific residuals require evaluation protocols that test not only whether generated code runs, but whether it faithfully implements the underlying scientific computation.