Uncovering the Latent Potential of Deep Intermediate Representations

Foundational models pretrained on huge amounts of data learn representations that evolve across depth, forming a hierarchy of embeddings with distinct semantic content and geometric structure. Contrary to the widespread practice of using only the final layer or shallow mixtures, we show that task-relevant information is distributed non-monotonically across layers and cannot be recovered by na\"ive aggregation. Through a geometric and empirical study across multiple modalities, we show that effective transfer depends on identifying which layers encode task-discriminative structure and how their embeddings are geometrically organized. We introduce Layer-wise Optimal Embedding Selection (LOES), a constructive spectral method that identifies task-discriminative subspaces by minimizing residual error under orthogonality and isotropy constraints. To align fine-tuning with this selection principle, we further propose Geometric Regularization (GeoReg), which enforces a simplicial structure on class manifolds and stabilizes representation geometry during fine-tuning. Across a wide range of architectures, depths, modalities, and data regimes, LOES consistently outperforms standard baselines, with gains that grow as model depth increases. Beyond accuracy, our method reveals how semantic factors are distributed across layers, thereby enabling cross-lingual and cross-modal interpretability analyses. Together, our results provide strong evidence that layerwise embedding geometry is not incidental but central to how deep models represent and transfer knowledge.

Uncovering the Latent Potential of Deep Intermediate Representations Modern AI models build understanding in stages, passing information through many internal layers. When people reuse a pretrained model for a new task, they almost always take only the final layer, assuming it holds the most useful information. But this is often the wrong choice: the most task-relevant information frequently sits in the middle layers, and no single layer holds all of it. We built a method called LOES that searches through every layer and picks the small handful that together carry the most useful, non-overlapping information for a given task. It does this quickly using a simple mathematical calculation rather than expensive retraining. We also added a companion technique that keeps these chosen layers well-organized when the model is further trained, preventing a common failure where the model's internal structure quietly degrades. Across images, text, speech, and combined data, our method consistently beats the standard approach, and the gains grow for larger models. It helps most for languages with little training data, such as Hindi, Arabic, and Urdu. Beyond accuracy, it reveals where different models store specific kinds of knowledge, giving researchers a clearer picture of how these systems actually work.