Quantifying the Effect of Test Set Contamination on Generative Evaluations

Test set contamination -- the inclusion of benchmarks in pretraining data -- is a critical threat to the trustworthy evaluation of AI systems. While its impact on discriminative evaluations is well-studied, contamination on generative evaluations remains underexplored. We quantitatively assess these effects across the language model lifecycle by pretraining models (up to 344M parameters) on web data contaminated with varying numbers of MATH test set replicas. Performance expectedly improves with contamination and model size, with improvements stemming from superficial memorization, not generalization. Our scaling law analysis reveals a fundamental breach: including even a single test set replica enables models to achieve lower loss than the irreducible error of training on the uncontaminated corpus. We then study additional training: overtraining with fresh data dilutes contamination effects, whereas supervised finetuning on the training set improves performance for low contamination but degrades performance for high contamination. At inference, we identify three distinct regimes of memorization—exponential decoherence to brittle memorization to deterministic lock-in—governed by solution length and sampling temperature. Finally, we identify and fix a critical implementation error in EleutherAI's ALM Evaluation Harness that previously underreported mathematical reasoning performance. By characterizing how generation and memorization interact, we highlight new considerations for trustworthy AI evaluation.