Abstract: We explore the conditions necessary to guarantee sharp upper bounds on the mean squared error when estimating mean counterfactual outcomes from observational data. In particular, we analyze the large family of designed-based weighting estimators which include balancing weights and matching. Beginning from the bias-variance decomposition, we argue that assumptions have to be made about the outcome function in order to choose a high performance estimator. For a theoretical framework, we use integral probability metrics and $\phi$-divergences to analyze the bias-variance trade-off. Finally, we consider conditions under which our mean squared error bounds are robust to failure of our assumptions.