Cross-Architecture CKA Reveals Where Sequence Models Converge and Diverge on Synthetic Mechanistic Tasks

Mechanistic interpretability has advanced primarily through architecture-specific case studies, particularly in Transformers, while recurrent models are typically compared only at the behavioral or formal-theoretic level. A basic empirical question therefore remains open: when matched-capacity sequence models are trained on the same controlled task, do they converge to aligned internal representations? We address this question with a compact benchmark comparing two-layer Transformers, LSTMs, and GRUs on five synthetic tasks spanning modular arithmetic, formal languages, and associative retrieval. For each architecture pair, we compute layer-wise centered kernel alignment (CKA) heatmaps averaged over eight random seeds, together with within-architecture seed baselines, a normalized convergence index, and layer-wise linear probes. Three findings emerge. First, recurrent models (LSTM, GRU) substantially outperform the Transformer under our fixed training protocol, creating an asymmetric-competence regime that must be accounted for when interpreting similarity scores. Second, recurrent–recurrent alignment consistently exceeds Transformer–recurrent alignment, particularly on hierarchical tasks (Dyck2), where the convergence index drops to 0.55– 0.68 across families versus 0.90 within the recurrent family. Third, moderate cross-architecture CKA persists even when all models perform near chance, indicating that geometric similarity can reflect shared heuristics or dataset structure rather than shared mechanistic solutions. These results suggest that representational convergence is taskcontingent rather than universal and that CKA is most informative when interpreted jointly with task performance, probing accuracy, and withinarchitecture baselines