Identifiable Equivariant Networks are Layerwise Equivariant

We investigate the relation between end-to-end equivariance and layerwise equivariance in deep neural networks. We prove the following: For a network whose end-to-end function is equivariant with respect to group actions on the input and output spaces, there is a parameter choice yielding the same end-to-end function such that its layers are equivariant with respect to some group actions on the latent spaces. Our result assumes that the parameters of the model are identifiable in an appropriate sense. This identifiability property has been established in the literature for a large class of networks, to which our results apply immediately, while it is conjectural for others. The theory we develop is grounded in an abstract formalism, and is therefore architecture-agnostic. Overall, our results provide a mathematical explanation for the emergence of equivariant structures in the weights of neural networks during training -- a phenomenon that is consistently observed in practice.

A common way to build neural networks that respect symmetries is to impose symmetry at every layer. This paper studies whether this layerwise structure is only a design choice, or whether it is also implied by equivariance of the whole network. Equivariance simply means that transforming the input leads to a transformation of the output, such as rotating an image and obtaining a rotated prediction. We show that, in many instances, an equivariant network can be represented in a form where its “active” layers are equivariant. In this sense, symmetry at the level of data enforce a symmetry inside the network.