Quaternion Self-Attention with Shared Scores

Quaternion neural networks are parameter-efficient and model multidimensional dependencies by representing four related features as a single entity. However, existing quaternion self-attention computes component-wise scores and applies independent softmax operations to each component, which increases the computational cost and allows attention distributions to diverge across components. We propose a shared-score quaternion self-attention mechanism that computes a single real-valued score using the quaternion inner product and applies a shared attention distribution across all components. This reduces score-computation multiplications by 75\% and the number of softmax operations from four to one. We prove that, when queries and keys are produced by quaternion linear projections that induce component pre-mixing, the component-wise and shared scores lie in the same interaction subspace, indicating that independent component-wise attention primarily re-parameterizes the same interactions rather than expanding the feature interaction space. In speech enhancement, our method reduces inference time by up to 44.3\% on a GPU and 58.1\% on a CPU while maintaining quality, with consistent trends across vision and natural language processing.

Modern AI systems often use a mechanism called attention to decide which parts of an input matter most. Quaternion neural networks are a specialized type of AI model that groups four related signals into one four-part unit, which can help preserve relationships such as those between different parts of a sound signal. Existing quaternion attention methods compute four separate attention scores for each unit. This is slower, and it is unclear whether the four scores provide genuinely different information. We show that, in the standard way these models mix the four parts before attention, the four scores mostly reuse the same underlying interactions. Based on this finding, we replace them with one shared score. This reduces the score calculation by 75% and keeps performance comparable on speech, image, and text tasks. The result makes quaternion attention simpler and faster, which may help deploy these models in settings where computation time matters, such as speech enhancement.