Flow Equivariant Transformers

Many sequences exhibit a geometric `flow' structure that both relates time steps within the sequence via a time-parameterized Lie-group action, and relates sequences to one another via the Lie algebra of the underlying flow. In this work, we illustrate how Transformers do not inherently respect this type of flow structure, causing otherwise related sequences to be processed as unrelated inputs, i.e., the same sequence viewed from different moving reference frames may be seen entirely differently. We then introduce a provably Flow Equivariant Transformer and demonstrate empirically that it achieves the generalization benefits expected of geometric inductive biases.