Convergence-Gated Distillation for Resource-Adaptive Reinforcement Learning Agents

Knowledge distillation is a core mechanism for adapting and compressing foundation models, but it is usually treated as a smooth process: better teachers should produce better students. We show that this assumption can fail sharply in self-supervised reinforcement learning. In contrastive RL, distilling from partially trained teachers produces nearly constant student performance across most of teacher training, even when teacher evaluation scores are already high. Student performance jumps only after the teacher crosses a convergence gate—a phase boundary in representation quality that determines whether its capability is transferable. We characterize this gate across depth-induced capability transitions in CRL. The gate is robust across three independently trained teachers and predictable from a zero-cost teacher-side behavioral diagnostic, but the diagnostic is not causal: shaping an unconverged teacher’s actions to mimic convergence does not open the gate. Instead, the mechanism is contrastive discrimination difficulty. Varying the InfoNCE temperature shifts the gate by 13 epochs, showing that the adaptation barrier is controlled by the self-supervised objective rather than by surface-level teacher behavior. Once the gate is crossed, distillation becomes an effective resource-adaptive mechanism. A width-scaled shallow student matches a deep teacher on Ant Big Maze and reaches 92% of teacher performance on Humanoid from a near-zero baseline. Equal-parameter controls show that shallow students can represent the capability but cannot discover it without a teacher. In contrast, SAC distillation transfers smoothly to a smaller student with no gate, reaching 96.6% of teacher reward. These results suggest that contrastive adaptation objectives can introduce sharp checkpoint-selection barriers absent from reward-based learning. For adaptive foundation-model pipelines, the practical implication is simple: which teacher checkpoint to distill from can dominate how much student capacity or training compute is required.