Sparse Autoencoders Can Learn Graded Latents for Relational Composition

Transformers trained on graph problems can compress structured sequences into fixed-size activation vectors. Prior work on a simple ordered bigram copying task presents a transformer model that does not use a separate feature for order; instead, it represents order through relative feature magnitudes. This raises a problem for interpreting sparse autoencoders (SAEs) trained on these activations: a latent may be meaningful because its value varies, not only because it is active. We train SAEs on this toy model and find graded latents whose activation values, not just their on/off status, correspond to different bigram orderings. In one SAE, one latent tracks how much more the model attends to the first input than the second input, with correlation 0.807; 98\% of latents have absolute correlation below 0.004. When we scale this latent, the model swaps 49.8\% of all outputs, or 77.8\% of outputs where the latent is active. These results suggest that SAE latents can encode relational information through magnitude, so analyses that reduce latents to feature presence indicators may misrepresent model features. Anonymised code is available at https://anonymous.4open.science/r/graded-latents-70FA.