DPMI: A Principled Index for Neural Polysemanticity via Dirichlet Process Mixture Modeling

Polysemanticity, where a single neuron responds to multiple unrelated concepts, is a central obstacle in mechanistic interpretability, yet the field lacks a principled continuous scalar for it. We introduce the $\textbf{Dirichlet Process Polysemanticity Index}$ (DPMI), a per-neuron score that combines inferred component count and component separation by fitting a non-parametric DPGMM and weighting by mean pairwise Jensen-Shannon divergence. On a controlled toy benchmark, DPMI achieves Spearman $\(\rho = 0.755\)$ and $AUROC \(= 0.877\)$, outperforming seven baselines. Against an independent Fourier-analytic ground truth from a modular-arithmetic transformer, DPMI remains significant with $\rho = 0.255\$ $\(p < 10^{-8}\)$. Across six architectures, we find a robust cross-modal law: language models are significantly more polysemantic than vision models $\(d = 0.803\$, $p < 10^{-129}\)$. Ablations show the non-parametric prior is essential (removing it drops $\(\rho\)$ by $0.040\-0.045\)$, and DPMI-guided SAE budget allocation improves reconstruction $\(R^2\)$ for the most polysemantic quartile by \(+0.010\) at fixed compute.