Quantifying and Optimizing Simplicity via Polynomial Representations

Deep networks often exhibit a preference for "simple" solutions, and such a simplicity bias is widely believed to play a key role in generalization. Yet a broadly applicable, quantitative measure of simplicity remains elusive. We introduce polynomial representations as a distribution-aware, low-dimensional surrogate for neural functions: we approximate a network’s predictive behavior along data-dependent interpolation paths using orthogonal polynomial bases, yielding a compact functional representation. We show that the effective degree of this representation serves as a practical simplicity metric that is predictive of generalization across tasks and architectures, and consistently outperforms existing generalization proxies such as sharpness. Finally, polynomial representations naturally yield a differentiable simplicity regularizer, which consistently improves generalization in image and text classification, fine-tuning contrastive vision–language models, and reinforcement learning.

Simplicity bias—the tendency of learning algorithms to prefer simpler solutions—is widely believed to be a key reason why deep networks generalize beyond their training data. Yet despite its importance, there is still no broadly useful way to measure or optimize the simplicity of a learned neural function. This work proposes a practical way to quantify simplicity by looking at how a model’s predictions change along interpolation paths between data examples. We approximate these prediction changes with low-dimensional polynomial curves and summarize their complexity using an “effective degree”: lower values mean the model behaves more smoothly and simply along the data-dependent paths. This measure is computed in function space rather than from the model’s parameters, making it broadly applicable across architectures and tasks. Across image classification, vision-language fine-tuning, and delayed generalization experiments, effective degree tracks generalization more reliably than common proxies such as sharpness and parameter norms. Because the polynomial fitting procedure is differentiable, the same idea can also be used as a training regularizer that discourages unnecessarily high-degree behavior. Adding this regularizer improves performance and robustness across image, text, and reinforcement learning benchmarks. Overall, the paper provides both a diagnostic tool for understanding neural network generalization and a practical method for training models that generalize better.