The Geometry of Updates: Fisher Alignment at Vocabulary Scale

Predicting transferability within shared-output model families (e.g., LLMs that share a vocabulary) poses a dilemma: representation-similarity metrics can be uninformative without assumptions about error geometry, while update-geometry metrics are computationally prohibitive. We show that, in a shared-output head setting, representation metrics (e.g., CKA) are non-identifiable for transfer; models can share identical representations yet have orthogonal head updates, so CKA alone cannot reliably rank transfer. We make head Fisher alignment tractable at vocabulary scale ($K{=}128{,}256$) using FisherSketch, a streaming algorithm that compresses joint (activation, error) geometry to a 16KB task signature ($m{=}4096$) with a 192KB per-task streaming state. Beyond the head, we prove a full-network decomposition and bounds, and we propose measurable diagnostics (profile cosine and off-diagonal discrepancy). We validate these on ViT-B/16 and LLMs up to 70B, showing that head/block approximations track the full Fisher and that FisherSketch remains informative in fixed-prefix verbalizer-shift settings where activation similarity cannot distinguish tasks.