The Geometry of Updates: Fisher Alignment at Vocabulary Scale

Training-free source selection for LLM families with shared vocabularies arises in scientific string domains such as SMILES, protein, and genomic sequences, where candidate corpora share a tokenizer but differ in prediction targets. This creates an activation-dark regime: representation-similarity metrics can be uninformative without assumptions about label-conditioned error geometry, while classical update-geometry metrics are computationally prohibitive at vocabulary scale. We show that, in a shared-output head setting, representation metrics (e.g., CKA) are non-identifiable for transfer; models can share identical representations yet have orthogonal head updates. The key identity is that head Fisher alignment is exactly a cosine between kernel mean embeddings in the joint activation-error space, exposing activation, error, and coupling factors rather than requiring a materialized Fisher matrix. FisherSketch estimates this cosine directly in a single streaming pass, making K=128,256 head Fisher alignment practical with a 16 KB task signature (m=4096) and a 192 KB per-task streaming state–small enough to store next to a model hash, but encoding transfer-relevant update structure. Beyond source selection, the same signatures and marginals provide a diagnostic instrument for studying whether LLM task similarity is driven by activations, errors, or their coupling; shared-parameter and internal-layer validations, together with Llama-3.1-8B verbalizer-shift experiments, show that FisherSketch remains informative when activation similarity cannot distinguish tasks.

Large language models are often improved by choosing existing data that will transfer well to a new task, especially in scientific strings such as molecule descriptions, proteins, or DNA, where many tasks use the same alphabet or vocabulary but differ in what the model must learn. Common shortcuts compare only the model’s internal activity, but this can be misleading: two tasks can look almost identical inside the model while requiring very different changes to its predictions. This paper introduces FisherSketch, a way to make a tiny fingerprint of a task that captures the direction in which the model would need to change, without storing huge records of all possible model changes. A central result is that this comparison can be turned into a simple question: do two task fingerprints point in similar directions? For models with very large vocabularies, a fingerprint can be only 16 KB, about the size of a short text file, and can be built in one pass over examples. The method helps choose useful source data and gives researchers a tool for studying whether tasks are similar because the model represents them similarly, because it makes similar prediction errors, or because of both.