Deep Trajectory Supervision: Deep Supervision Strikes Back

Interpreting residual architectures as discretizations of Ordinary Differential Equations characterizes the forward pass as a continuous latent flow. Although this framework defines the mechanics of inference, conventional training paradigms primarily constrain the terminal state, leaving the intermediate evolution unregulated. In this work, we formalize the forward pass as a Conditional Discriminative Flow and investigate its intrinsic kinematic laws. Using Tuned Lens analysis, we discover that the accumulation of semantic evidence follows a consistent exponential schedule. This finding confirms that deep models naturally require an extended phase of feature construction prior to a rapid transition toward categorical certainty in the terminal layers. Standard training ignores this latent progression. To resolve this impedance mismatch, we propose Deep Trajectory Supervision, a framework that aligns auxiliary supervision with this intrinsic exponential bias. By rectifying the trajectory of the inference flow, DTS functions as a critical physical inductive bias. Empirical evaluations on ImageNet-1K and various benchmarks demonstrate that DTS significantly accelerates convergence and improves terminal performance.

Modern image-recognition systems are usually trained only to make the correct final prediction, but we rarely guide how their internal understanding should develop from early to late layers. This can be inefficient, because early layers may be pushed to make confident decisions before they have built useful visual features. We study this internal process by measuring how much evidence for the correct class appears at each depth of a deep network. Across several models and datasets, we find a consistent pattern: this evidence grows slowly at first and then rises quickly near the end of the network. Based on this observation, we propose Deep Trajectory Supervision, a training method that gives intermediate layers soft targets whose confidence increases with depth, instead of forcing every layer to predict a hard label. This makes the training signal better match how deep networks naturally build semantic information. In experiments on standard image-classification benchmarks, including ImageNet, our method improves accuracy and helps models reach target performance faster, while adding little training cost and no inference-time cost. More broadly, our work suggests that not only the final answer, but also the path a model takes to reach that answer, is important for training better neural networks.