Tailoring Strictly Proper Scoring Rules for Downstream Tasks: An Application to Causal Inference

Probabilistic models are typically trained using task-agnostic objectives like log-loss, which can lead to significant errors in downstream estimation. This disconnect is especially critical in Inverse Probability Weighting (IPW) for causal inference, where propensity score errors near $0$ and $1$ often lead to high bias and variance. We propose a principled framework for deriving task-specific strictly proper scoring rules by matching the local curvature of the downstream error metric. We apply this to the Average Treatment Effect (ATE) estimation, deriving a closed-form loss and its corresponding canonical probability mapping that can be readily integrated with any model like a neural network or a gradient boosting algorithm. Extensive evaluations on causal inference benchmarks demonstrate that our tailored objective consistently outperforms standard likelihood-based and covariate-balancing approaches.

To see if a medical treatment works using past records, we need fair comparisons. Because sicker patients often get the drug, simply comparing who took it and who didn't biases the results. To fix this, math gives extra "weight" to rare patients. If a healthy person takes the drug, their data is multiplied, based on the inverse of their probability of getting the treatment, to represent all the healthy people who didn't. Usually, machine learning models are used to estimate this probability. However, machine learning models aren't perfect. If a model makes a tiny error for an unusual patient, for example, estimating a 0.1% chance instead of 1%, their mathematical weight explodes, ruining the calculation. Standard training treats this catastrophic error the exact same as a harmless one. To deal with this, we created a new training rule based on the final goal: correctly estimating the treatment effect. We designed a training rule that punishes the model's errors exactly as much as those errors distort the final answer. When tested, this custom training gave more accurate treatment effect estimation and it is a simple upgrade that plugs directly into existing machine learning models.