Conditional Dependence Structure in Sparse Autoencoder Features

Sparse Autoencoders (SAEs) decompose language-model activations into large overcomplete dictionaries of interpretable features, but these features are typically used as individual directions rather than as structured representations. We examine whether SAE features exhibit recoverable conditional dependence. We introduce a scalable estimator combining support-based screening, nodewise LASSO, and resampling-based null calibration to construct dataset-conditioned graphs over SAE features. Applied to GemmaScope SAEs on FineWeb and WMDP-bio, the graphs produced by this approach are sparse, stable under resampling, and modular, with many local neighborhoods linking semantically related features. These graphs also show shared cross-dataset and corpus-specific organization. They are not well explained by decoder cosine similarity or raw activation correlations. These results suggest that SAE representations contain organization beyond individual features, providing a way to study feature splitting, merging, and organization in overcomplete representations.