Timezone: »

Positional Encodings as Group Representations: A Unified Framework
Derek Lim · Hannah Lawrence · Ningyuan Huang · Erik Thiede
Event URL: https://openreview.net/forum?id=18f4nhMJ33 »

Positional encodings are ubiquitous as an input featurization tool in language modeling, computer vision, and graph representation learning, enabling neural networks to capture important geometric structure of the input. Traditionally, positional encodings have been defined anew for each data domain. In this work, we reinterpret positional encodings for disparate data types --- including sequences, grids, graphs, and manifolds --- in the unifying framework of group representations. We show how to express existing positional encodings as group representations, and conversely, propose new positional encodings by choosing suitable groups and representations. We validate our framework with experiments on implicit neural representations of images and vector fields, highlighting the practical utility of such positional encodings for encouraging approximate equivariance and capturing geometric structure.

Author Information

Derek Lim (MIT)
Hannah Lawrence (Massachusetts Institute of Technology)
Ningyuan Huang (Johns Hopkins University)
Erik Thiede (Flatiron Institute)

More from the Same Authors