Algorithmic predictions are emerging as a promising solution concept for efficiently allocating societal resources. Fueling their use is an underlying assumption that such systems are necessary to identify individuals for interventions. We propose a principled framework for assessing this assumption: Using a simple mathematical model, we evaluate the efficacy of prediction-based allocations in settings where individuals belong to larger units, such as hospitals, neighborhoods, or schools. We find that prediction-based allocations outperform baseline methods using aggregate unit-level statistics only when between-unit inequality is low, or the intervention budget is high. Our results hold for a wide range of settings for the price of prediction, heterogeneity of treatment effects, and learnability of unit-level statistics. Combined, we highlight the potential limits to improving efficacy of interventions through prediction.