Platonic Transformers: A Solid Choice For Equivariance

While widespread, Transformers lack inductive biases for geometric symmetries common in science and computer vision. Existing equivariant methods often sacrifice the efficiency and flexibility that make Transformers so effective through complex, computationally intensive designs. We introduce the Platonic Transformer to resolve this trade-off. By defining attention relative to reference frames from the Platonic solid symmetry groups, our method induces a principled weight-sharing scheme. This enables combined equivariance to continuous translations and Platonic symmetries, while preserving the exact architecture and computational cost of a standard Transformer. Furthermore, we show that this attention is formally equivalent to a dynamic group convolution, which reveals that the model learns adaptive geometric filters and enables a highly scalable, linear-time convolutional variant. Across diverse benchmarks in computer vision (CIFAR-10), 3D point clouds (ScanObjectNN), and molecular dynamics, property prediction and generation (OMol25, ProteinMD, QM9), the Platonic Transformer achieves competitive performance by leveraging these geometric constraints at no additional cost.

Many important machine learning problems involve objects with geometry, such as molecules, 3D shapes, and images. For example, a molecule has the same chemical properties no matter how it is rotated in space, but standard Transformers do not naturally understand this kind of symmetry. Existing symmetry-aware models can handle these transformations, but they often require specialized and expensive designs. We introduce the Platonic Transformer, a way to give standard Transformers a built-in sense of geometric symmetry without changing their basic structure or increasing their computational cost. The idea is to view data through reference frames inspired by highly symmetric shapes, such as tetrahedra and octahedra, and to share model weights across these views. This helps the model make consistent predictions when inputs are moved or rotated. Across image, 3D object, and molecular tasks, our model performs competitively while staying as efficient and flexible as a standard Transformer. This makes symmetry-aware machine learning more practical for scientific applications such as molecular property prediction, molecule generation, and force prediction.