Dynamic Relational Priming Improves Transformer in Multivariate Time Series

Standard attention mechanisms in transformers employ static token representations that remain unchanged across all pair-wise computations in each layer. This limits their representational alignment with the potentially diverse dynamics of each token-pair interaction. While they excel in domains with relatively homogeneous relationships, standard attention may be inadequate in capturing heterogeneous inter-channel dependencies of multivariate time series (MTS) data where different channel-pair interactions within a single system may be governed by entirely different physical laws or temporal dynamics. To better align the attention mechanism for such domain phenomena, we propose attention with dynamic relational priming (prime attention). Prime attention modulates token representations for each token-pair, optimizing each pair-wise interaction for that specific relationship. Our results demonstrate that prime attention consistently outperforms standard attention across benchmarks, achieving up to 6.5\% improvement in forecasting accuracy. In addition, prime attention achieves comparable performance using up to 40\% less sequence length compared to standard attention, demonstrating its superior relational modeling capabilities and potential for data efficiency.

Transformers, the technology behind modern AI systems, rely on a mechanism called "attention" that lets different pieces of data interact and share information. In standard attention, each piece of data presents the same fixed version of itself to every other piece it interacts with. This works well when all the relationships in the data are similar in nature, as in language. But many real-world systems aren't like that. In multivariate time series such as weather sensors, power grids, or traffic networks, different pairs of signals can relate through completely different underlying dynamics. We introduce prime attention, which lets each piece of data adjust how it presents itself depending on which partner it is interacting with, tailoring every pairwise interaction to its specific relationship. Across 11 real-world forecasting benchmarks, this consistently improves accuracy over standard attention and other competing methods, while also learning more efficiently from less data.