Needles in the Haystack: Addressing Signal Dilution Improves scRNA-seq Perturbation Response Modeling and Evaluation

Recent benchmarks reveal that single-cell perturbation response models are often outperformed by simply predicting the dataset mean. Through large-scale *in silico* simulations, together with analyses of two real-world perturbation datasets, we trace this anomaly to a metric artifact: unweighted error metrics systematically reward mean predictions when perturbation effects are sparse. To address this limitation, we introduce differentially expressed gene (DEG)-aware metrics—weighted mean-squared error (WMSE) and weighted delta $R^{2}$ ($R^{2}_{w}(\Delta)$)—that sensitively measure error in niche, perturbation-specific signals. We further propose explicit negative and positive performance baselines to calibrate these metrics. Under this framework, the mean baseline sinks to null performance, while genuinely informative predictors are correctly rewarded. Finally, we show that using WMSE as a training objective reduces mode collapse and improves predictive performance across multiple model architectures.

The ability to reliably predict how cells respond to perturbations in the lab could save years and resources in the drug development pipeline. However algorithms designed for this purpose currently face controversy: several high profile methods claim outstanding performance, while independent benchmarking finds their results worse than simply predicting an average. In our paper, we explore this discrepancy and track the issue to unexpected behaviours of common metrics when applied to high dimensional gene expression data. We find a main culprit on a needle in the haystack problem: only a tiny fraction of genes changes meaningfully under a perturbation. This means that an average prediction is actually correct for the vast majority of genes but is wrong in the ones that matter. Current metrics don't account for this and hence reward statistical over biological estimation. Using these insights, we propose a novel standard protocol for evaluating model performance that includes two new niche-sensitive metrics along with positive and negative controls. We then realized one of these metrics could be repurposed as a training signal. Under this new supervision, models improved with respect to their original training when evaluated on independent metrics. This finding was replicated across multiple method families suggesting that focusing on strong niche signals is an effective strategy for learning how cells respond to perturbations.