ConfPert: Distribution-Free Conformal Coverage for Single-Cell Perturbation Predictors

Single-cell perturbation predictors are typically evaluated on mean-prediction metrics, but recent work shows that deep-learning foundation models systematically underperform a simple bilinear ridge baseline on these metrics (Ahlmann-Eltze et al. 2025; Csendes et al. 2025), and no existing method provides coverage guarantees on the distributional fit between predicted and observed cell populations. We introduce ConfPert, a model-agnostic conformal prediction framework that gives finite-sample coverage guarantees on six per-population distributional discrepancies, exposed at four head levels: per-gene, per-perturbation, per-population, and subgroup-conditional. We use ConfPert to evaluate eight perturbation predictors spanning $0$ to $\sim 6 \times 10^8$ parameters across five single-cell datasets, including a cross-cell-line transfer split. The protocol is pre-registered: hypotheses, permutation-test thresholds, and multiple-comparisons corrections are committed to git before any first run. We find three things. First, distributional-output predictors with sparse-additive structure recover bimodal cell-fate structure that mean-only predictors cannot, achieving near-exact nominal coverage on the variance-sensitive metrics. Second, the pre-registered capacity-hurts hypothesis fails the overall threshold but passes on the non-cancer cell line, identifying cell-line context (not model size alone) as the load-bearing covariate for whether scaling improves calibration. Third, on real PRISM Repurposing drug-screen data, calibrated signatures recover more Benjamini-Hochberg-corrected drug-resistance pathways than uncalibrated baselines, a result robust across signature-size choices. We release a pip-installable confpert library and a Cell-Eval plug-in that fills a gap in Arc Institute's evaluation backbone.