Timescale Separation in Sparse Dictionary Learning: Reconstruction Converges Before Reproducibility
Bright Liu
Abstract
Sparse autoencoders (SAEs) trained on neural network activations reach low reconstruction loss within tens of epochs, yet cross-seed decoder reproducibility—measured by mean maximum cosine similarity (MMCS) calibrated against an extreme-value-theory random-dictionary null—remains indistinguishable from random dictionaries on the same timescale. We document this timescale separation in a mod-113 algorithmic transformer ($d=128$) and validate it across Pythia-70M, 160M, and 410M ($d=512$–$1024$). On mod-113, reconstruction loss saturates by epoch 20 while TopK decoder MMCS continues climbing for 1,000+ epochs, asymptoting at 0.73 (2.44× random). On Pythia-410M layer 21, a $d_\text{sae}$ sweep shows that all overparameterization ratios ($d_\text{sae}/\text{erank}$ from 1.3 to 10.0) converge to 4.3–4.7× random given sufficient training, dissolving the apparent $d_\text{sae}/\text{erank}$ stability threshold. We derive this null from the order statistics of inner products between random unit vectors, obtaining predictions within 0.03% of empirical baselines across dimensions 128–1024. Dead-neuron resampling produces a transient MMCS spike to 0.888 at epoch 10 that decays below the unregularized baseline—a phantom consistency artifact arising from training dynamics rather than learned structure.
Chat is not available.
Successful Page Load