Where to Read a Frozen Audio Encoder: Objective-Induced Geometry and Zero-Label Layer Selection

Pretrained audio encoders are increasingly reused across speech, environmental sound, music, and paralinguistic tasks, yet frozen-feature pipelines almost always read out the final layer by default. We show this default is often a major failure mode: across seven encoders, Whisper-S, CLAP-HTSAT, HuBERT-B, WavLM-B, UniSpeech-SAT, wav2vec2-B, and Data2Vec-Audio, and four audio domains, environmental sound (ESC-50), urban sound (UrbanSound8K), music genre (GTZAN), and speech emotion (CREMA-D), final-layer extraction can silently discard 24 to 38 score points relative to an earlier depth. The pretraining objective induces a predictable representation geometry across depth, and that geometry localizes the useful extraction layer well enough to select it with zero target-task labels: choosing the layer with highest average normalized isotropy and participation ratio improves character error rate on 11 of 12 low-resource ASR settings, a 17.0\% average relative reduction in CER and 23.9\% for wav2vec2-large across six languages. A counterintuitive regime is that low isotropy is useful when class scatter aligns with the low-rank objective subspace, so hard-unit and speaker-aware SSL encoders break the global ``more isotropy is better'' rule in a predictable way. \textsc{QuickLayer}, a weighted rule that adds a few-shot probe to the geometry score, recovers 83 to 89\% of the avoidable last-layer gap, turning extraction depth into a measurable, label-cheap choice for any frozen audio pipeline.