CausalProfiler: Generating Synthetic Benchmarks for Rigorous and Transparent Evaluation of Causal Machine Learning

Causal machine learning aims to answer "what if" questions using machine learning algorithms, making it a promising tool for high-stakes decision-making. Yet, empirical evaluation practices remain limited. Existing benchmarks often rely on a handful of hand-crafted or semi-synthetic datasets, leading to brittle, non-generalizable conclusions. To bridge this gap, we introduce CausalProfiler, a synthetic benchmark generator for causal machine learning methods. Based on a set of explicit design choices about the class of causal models, queries, and data considered, CausalProfiler randomly samples causal models, data, queries, and ground truths constituting the synthetic causal benchmarks. In this way, causal machine learning can be rigorously and transparently evaluated under a variety of conditions. This work offers the first random generator of synthetic causal benchmarks with coverage guarantees and transparent assumptions operating on the three levels of causal reasoning: observation, intervention, and counterfactual. We demonstrate its utility by evaluating several state-of-the-art methods under diverse conditions and assumptions, both in and out of the identification regime, illustrating the types of analyses and insights CausalProfiler enables.

Causal machine learning aims to answer questions about causes, interventions, and alternative outcomes, but evaluating these methods is difficult: in real life, the ground truth is often unknown or impossible to observe, and experiments that could provide evidence may be expensive or unethical. Existing evaluations, therefore, rely on limited real-world studies or a small number of synthetic datasets, which can hide important assumptions and give an incomplete picture of performance. We introduce CausalProfiler, an evaluation framework that generates diverse synthetic causal benchmarks under clearly specified assumptions. Researchers define the kinds of settings, data, and questions they want to study, and CausalProfiler generates benchmarks for evaluating methods under these conditions. This makes it possible to evaluate causal methods systematically, including when their assumptions hold and when they are violated. Rather than reporting performance on a few selected datasets, researchers can investigate broader patterns of success, failure, and sensitivity to assumptions. Synthetic evaluation cannot replace real-world evidence, but CausalProfiler provides a rigorous and transparent framework for evaluating causal machine learning methods before they are relied upon in practice.