NumLeak: Public Numeric Benchmarks as Latent Label in Foundation Models

Foundation-model evaluations increasingly rely on public benchmark datasets whose numeric values appear in pretraining: financial factor returns, macroeconomic releases, climate records. If models recover these historical values from a date alone, evaluations that look out-of-sample may instead measure memorized benchmark access. We introduce NumLeak, a measurement framework pairing API-boundary measurement on production models with white-box controlled validation on an open causal LM. Across nine frontier LLMs and three public benchmark domains, top tier models recall the Fama–French market excess return at Pearson r=0.92–0.99 selectively over five other factors in the same library, with weaker recall on smaller and non-frontier tiers; the channel extends to U.S. unemployment, CPI inflation, and NOAA temperature. Post-2025 months collapse to 21–57% parse rate while recall stays at r≈0.99 on the parsed subset, the asymmetry expected from a memorization channel rather than generic numeric fluency. A soft one-line preamble closes 99.8% of attack attempts at near-zero utility cost on conceptual and qualitative-historical finance queries. Code: https://anonymous.4open.science/r/numleak-656C.