The Compositional Generalization Gap in Named Entity Recognition: Static Benchmarks Overestimate Transferable Performance

Static benchmarks remain the dominant tool for evaluating named entity recognition (NER), yet they may systematically overestimate transferable performance under compositional distribution shift. We report a 38–49 point F1 drop when fine-tuned NER models are evaluated on novel distributional variants of CoNLL-2003: models that score 89–92% F1 on the static benchmark fall to 39–43% on perturbations that recombine entity types, contexts, and surface forms in ways unseen during training. Across 12 models spanning fine-tuned transformers, open-source LLMs (3B–14B parameters), and frontier API models, we observe an average gap of +30.5 points; trained human annotators reach 77.8% F1 on the same compositional instances, leaving a 35.8 point human-model gap. The structure of the gap is consistent with a compositional generalization failure: models acquire benchmark-specific bindings between entity surface forms and contexts rather than transferable compositional structure. We isolate which compositional perturbations are most damaging (LOC +42.7 pp > PER +29.5 pp > ORG +14.1 pp > MISC +0.2 pp), show the gap correlates strongly with naturally occurring out-of-distribution difficulty (WNUT-2017, r = 0.916), and verify that it persists under contamination-detection screening, prompt ablations, and a clean-subset control. We argue that compositional evaluation protocols are necessary for any reliability claim about NER systems and release a perturbation suite, the Compositional Evaluation Engine (CEE), that can be applied to any information-extraction benchmark.