Inducing Causal Order through Tabular In-Context Learning

Tabular foundation models (TFMs) achieve state-of-the-art predictive performance on tabular data, but primarily rely on correlational structure susceptible to change under distribution shift or intervention. To address this, we propose inducing causal structure into TFMs by constraining predictive attention to features that precede a given target under a causal ordering. Our architecture differentiably ranks tabular columns and constructs an attention mask that restricts information flow in the anticausal direction. Crucially, we infer the causal order itself directly from data by likelihood maximization without access to the underlying causal structure during training. Experiments across diverse classes of structural causal models show that our method TabOrder reliably infers accurate causal orderings, enabling it to outperform purely associational models under interventions and distribution shift.