PACEAttention: Principled and Adaptive Feature Compression-Expansion Grounded in the Geometry of $\text{MCR}^2$

The maximal coding rate reduction ($\text{MCR}^2$) objective is proposed for learning low-dimensional subspace representations and for principled deep model design, where layer structures are derived by unrolling its optimization steps. However, existing methods motivated by this objective do not fully adhere to design principles implied by the $\text{MCR}^2$ gradient, which weakens the principled and interpretable foundations of the resulting models. In this work, we introduce PACEAttention, a novel principled attention mechanism inspired by the *geometric insight* of $\text{MCR}^2$, whose gradient-based updates move features along directions shaped by the underlying low-dimensional feature structure. Our method captures this structure by leveraging randomization to guide feature updates. This principled construction enables the resulting PACENet to exhibit enhanced interpretability, with different heads attending to distinct image regions and capturing *fine-grained* structures under simple supervised training. Experiments demonstrate that PACEAttention achieves superior performance and more stable scalability than previous principled modules while remaining low complexity.

Deep neural networks are often powerful but difficult to interpret, making it unclear why they work well or how their architectures should be designed. In this work, we propose PACEAttention, a new attention mechanism motivated by geometric principles from representation learning theory. Our method entails feature learning as a progressive process, where same-category feature representations are compressed into compact subspaces, while cross-category representations are expanded. This process is guided by the intrinsic structure of the data. The resulting model, PACENet, is both principled and efficient, and produces competitive performance on several classification benchmarks. This design also makes PACENet easier to interpret, with different parts of the model naturally focusing on different image regions and fine details, even when trained using standard supervised learning.