SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training

Many machine learning applications rely on heterogeneous event streams to make predictions, either causally as events arrive or bidirectionally over complete sequences. We propose SOHET (Sequence Of Heterogeneous Events Transformer), a hierarchical architecture combining event-type-specific tabular encoders with temporal and type embeddings, processed by a causal or bidirectional transformer. We introduce three self-supervised pre-training objectives for the causal setting. On a proprietary large-scale real-world fraud detection task with 17 event types, SOHET outperforms FlexTPP, NAPPT, and CIPPT by 6.6%. Pre-training yields an additional 5.9% gain and 20× faster convergence. On the EBES benchmark, bidirectional SOHET achieves state-of-the-art on 6 out of 8 tasks.