$\text{DT}^\text{2}$: Decision-Targeted Digital Twins

A digital twin (DT) is a virtual model of a real-world system that can assist decision-making by simulating scenarios induced by different policies. However, typical machine learning-based DTs do not optimise for this use case. We prove that, when model capacity is limited, training DTs to minimise one-step transition errors can produce suboptimal models for ranking sets of policies according to a reward function. We further show that this holds empirically, even with expressive model classes. To address this, we introduce DT$^2$, a decision-targeted DT training paradigm. Firstly, DT$^2$ uses fitted Q-evaluation to estimate values of candidate policies from offline data. A DT is then trained to generate rollouts that preserve pairwise policy rankings derived from these proxy ground-truth values with an architecture-agnostic loss function. We empirically demonstrate the efficacy of our method across a range of settings and architectures. DT$^2$ consistently improves policy ranking and reduces decision regret during policy selection relative to conventional DT training, both for policies used during training and for unseen policies, while maintaining a good level of raw simulation fidelity.

This work introduces a new way to build “digital twins” — virtual simulators of real-world systems to help people compare different decisions or strategies. Current digital twins are usually trained to imitate observed past data as accurately as possible, but this does not always help them identify the best decision. We propose a method that instead trains digital twins to lead to better decisions, while still producing realistic simulations. Across several experiments, including medical treatment planning, the approach better identifies more effective decisions compared to existing methods.