How Should Transformers Encode Numeric Values in Electronic Health Records?

How do we encode numeric values in transformer-based sequence processing, particularly in electronic health record (EHR) data? We systematically compare discrete, continuous, and hybrid value encoding strategies using synthetic arithmetic tasks embedded within real-world EHR data, as well as real-world clinical prediction tasks. Our study reveals trade-offs between numeric precision, optimisation stability, and architectural flexibility. We find that approaches that explicitly model value-concept interactions perform best on precision-sensitive arithmetic tasks when architectural constraints permit. Hybrid token-based approaches that retain numeric values but apply binning prior to projection provide a more robust and broadly applicable alternative, with the optimal number of bins following a simple empirically derived power-law in dataset size. Across tasks, models consistently exhibit reliable “good enough” numeric computation rather than exact arithmetic, while clinical gains from incorporating laboratory values are task-dependent. This suggests that robustness and deployability often outweigh maximal numeric precision in practice, motivating hybrid token-based approaches as a practical default.

Electronic health records (EHRs) contain many numeric measurements, such as laboratory test results, but there is no clear consensus on which encoding strategies work best for different tasks, or when they fail. This makes it difficult to compare methods and choose robust approaches for EHR transformer models. We developed a test suite for evaluating numeric encoding strategies in EHR transformers, using both controlled synthetic tasks and real clinical prediction tasks. The synthetic tasks are embedded in real patient histories, allowing us to evaluate numeric reasoning under clinically relevant conditions, including precise arithmetic, trend detection, and optimisation stability. We find that most methods can perform “good enough” arithmetic rather than exact calculation. Joint embedding methods work best when fine-grained numeric precision is required, while hybrid token-based methods provide a robust all-around approach that is easier to scale and deploy.