Black-Box Assisted Regression: Phase Transitions and Minimax Optimality

Foundation models are often used as fixed black-box predictors for downstream tasks with limited labeled data, but their predictions may be biased and unsafe to trust blindly. We study this setting through black-box assisted nonparametric regression: a learner observes labeled samples and can query a fixed predictor $f_0$, while the target $f^*$ is close to $f_0$ in $L_2(P_X)$ up to an unknown radius $\delta$. We give a finite-sample minimax characterization showing a phase transition at $\delta_c(n)\asymp n^{-\beta/(2\beta+d)}$, with leading risk $\min\{\delta^2, n^{-2\beta/(2\beta+d)}\}$. We then analyze a Safe Residual Estimator: it learns a correction around $f_0$, initializes the residual head at zero so the initial predictor equals $f_0$, and uses holdout selection to revert to $f_0$ when the learned correction is not supported by validation data. Here, ``safe'' means avoiding negative transfer, i.e., performing worse than the black-box predictor alone. The estimator matches the leading minimax term up to an additive validation-selection cost. Synthetic regression experiments verify the predicted phase transition, while CIFAR-100 with CLIP and AG News with Qwen3-8B provide practice-facing evidence that the same residual-correction tradeoff is useful beyond the formal squared-loss regression setting.

Modern AI systems, such as large language models and vision models, are often used as black-box predictors for new tasks. These models can be very helpful when only a small amount of labeled data is available, but their predictions may also be biased or unreliable for the specific task at hand. This paper studies when and how such a black-box model should be trusted. We show that there is a clear boundary between two situations: when the black-box model is already accurate enough, it is best to stay close to its predictions; when it is less accurate, the available labeled data should be used to correct it. Based on this idea, we propose a simple and safe correction method. The method starts from the black-box prediction, learns a correction from labeled examples, and uses a validation set to decide whether the correction is actually helpful. If the correction does not improve performance, the method falls back to the original black-box model. Our theoretical results explain this tradeoff and show that the proposed method achieves near-optimal performance in the studied regression setting. Experiments on synthetic data, image classification with CLIP, and text classification with Qwen3-8B support the theory and show that this safe correction strategy can improve sample efficiency while reducing the risk of making the black-box model worse.