Uncovering Bias Mechanisms in Observational Studies

Observational studies are a key resource for causal inference but are often affected by systematic biases. Prior work has focused mainly on detecting these biases, via sensitivity analyses and comparisons with randomized controlled trials, or mitigating them through debiasing techniques. However, there remains a lack of methodology for uncovering the underlying mechanisms driving these biases, e.g., whether due to hidden confounding or selection of participants. In this work, we show that the relationship between bias magnitude and the predictive performance of nuisance function estimators (in the observational study) can help distinguish among common sources of bias. We validate our methodology through extensive synthetic experiments and a real-world case study, demonstrating its effectiveness in revealing the mechanisms behind observed biases. Our framework offers a new lens for understanding and characterizing bias in observational studies, with practical implications for improving causal inference.

Many healthcare questions cannot be answered using randomized clinical trials alone, because trials can be expensive, slow, or limited to specific patient groups. Researchers therefore often use existing medical records or insurance claims to study whether treatments help or harm patients. These studies can cover many more people, but they can also give misleading answers when the people who receive a treatment differ from those who do not, or when the study includes an unrepresentative group of patients. This paper develops a method for understanding why an observational study may disagree with a randomized trial. Instead of only asking whether the observational study is biased, we ask what kind of bias is most likely responsible. By testing our method in simulations and in data from the Women’s Health Initiative, we show that it can provide useful clues about the source of disagreement. This can help researchers design better studies and interpret real-world medical evidence more carefully.