From Teacher Pathways to Invariant Manifolds: Consensus Subspace Distillation for TSFMs

Time-series foundation models (TSFMs) deliver strong cross-domain generalization, but their scale makes deployment costly. Knowledge distillation is a natural compression route, yet prior TSFM distillation typically imitates teacher outputs, features, or pairwise relations, and therefore remains tightly coupled to teacher-specific training trajectories while underutilizing two empirical properties: (i) high-level representations across model scales tend to converge toward a shared, approximately low-rank geometry, and (ii) layer-wise utility follows a long-tail pattern. We propose consensus subspace distillation, which reframes distillation as aligning a student to a model-agnostic geometric object: a scale-invariant low-rank consensus subspace together with its center statistics. Offline, we screen high-contribution layers via drop-layer marginal loss, estimate a shrinkage-stabilized covariance from their embeddings, and derive a truncated eigensubspace that defines a consensus projector. Online, we project student embeddings into this subspace and match the teacher’s projected mean and covariance using a lightweight mean--covariance objective, enabling stable optimization without rigid pointwise feature binding. To mitigate subset-induced bias, we further introduce a frequency-domain uncertainty injection mechanism that inflates spectral density based on characteristic-function discrepancies and injects dispersion only within the consensus directions. Across forecasting and imputation, the distilled student matches or slightly improves upon the teacher, while exhibiting a predictable trade-off under strict zero-shot classification. With MOMENT-Large as teacher, we achieve about 90% parameter reduction and substantial distillation-time savings while retaining comparable performance across multiple time-series tasks. Code and compressed weights are available at anonymous.4open.science/r/CSD-13C3/.

Time-series data are measurements recorded over time, and they appear in many important settings, such as healthcare signals, electricity demand, and economic indicators. Large AI "foundation models" can make strong predictions across all of these, but they are often too big and costly to run on devices with limited computing power. A common way to shrink them is to train a small "student" model to imitate a large "teacher" model, but existing methods imitate it too literally and miss its broader, more useful structure. We found that models of very different sizes tend to settle on the same underlying internal structure, much like maps at different scales that still agree on the major landmarks. Instead of forcing the student to match every detail, we teach it to capture this shared structure directly, and we account for the uncertainty of compressing on only part of the data, helping the student generalize. The resulting model is far smaller, and the compression process itself is much faster, while still matching the teacher on key tasks such as forecasting and filling in missing values. This makes powerful time-series AI far easier to deploy in practical, resource-limited settings such as phones, sensors, and edge devices.