The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms

Traditional evaluations measure a learning algorithm's final performance on an i.i.d. test set, reducing learning to a single aggregate score. This approach obscures a fundamental question: to what extent does learning from a specific example generalize to others? Such per-sample generalization—akin to learning by analogy in human cognition—captures how far the knowledge extracted from one example can transfer, yet remains invisible to standard benchmarks. We introduce the Generalization Spectrum, an evaluation framework designed to expose this hidden dimension. For each training example, we construct a controlled suite of test variants arranged by increasing transfer distance—from exact recall to implementation transfer across languages, context transfer under complete narrative re-framing, category-matched in-domain problems, and an unpaired baseline. By tracking performance across these distances, we reveal not just whether an algorithm learns, but how far that learning extends. We instantiate this framework on competitive programming, using a synthetic generation pipeline seeded with recent problems to mitigate contamination. We first compare three canonical learning paradigms under matched memorization. RL converts memorization into near-transfer more efficiently than SFT-family baselines, while ICL exhibits strong but correspondence-dependent transfer. We then use the Spectrum to diagnose within-family variants. The resulting profiles show that local gains need not expand the generalization radius: abstractions and hints mainly lift local transfer, RFT preserves a stronger far-transfer tail than reference SFT, and self-distillation or hint-assisted RL can reduce far transfer even when local transfer or optimization improves.

Modern AI systems can improve after seeing examples, but standard benchmarks usually report only one final score. This makes it hard to tell whether a system truly learned a reusable idea, simply memorized the example, or can only handle very similar cases. This paper introduces the Generalization Spectrum, a way to measure how far learning from one example transfers to related problems. For each training problem, we create several test problems at increasing distances: the original problem, the same logic in another programming language, the same solution hidden in a different story, a related problem from the same category, and an unrelated problem from the same domain. We use competitive programming tasks because answers can be checked automatically and problem structure can be varied in controlled ways. Our experiments compare in-context learning, supervised fine-tuning, and reinforcement learning. We find that different learning methods have very different transfer patterns, even when they appear similar under aggregate scores. Reinforcement learning tends to convert memorization into nearby transfer more effectively than supervised fine-tuning, while in-context learning works well when the relevant example can be clearly matched to the new problem but drops when that connection becomes weaker.