Beyond Point Predictions: Manifold Expansion and Dual Alignment for Robust Time Series Distillation

Knowledge Distillation (KD) promises to bridge the gap between the high computational costs of Transformer-based models and the expressiveness limitations of linear models in long-term time series forecasting. Many existing time series distillation methods inherit the computer vision paradigm, constraining student models by minimizing point-wise prediction matching (output-level distillation) errors. However, blindly mimicking teacher predictions, which can be uncertain, may induce negative transfer. To address this, we propose Dynamic Structural Distillation (DSD), a robust framework that goes beyond the prediction-matching paradigm. First, we design LMP-Net, leveraging manifold expansion to project features into a high-dimensional latent space, alleviating the expressiveness bottleneck while preserving lightweight inference. Second, to address token–point architectural mismatch, we propose Dual Manifold Alignment, employing Similarity-Preserving Knowledge Distillation (SPKD) and Optimal Transport (OT) to align features at the topological and geometric levels, respectively. Finally, we introduce Regime-Aware Adaptive Distillation (RAAD) to reduce the risk of teacher misguidance via a dataset-level regime prior and a confidence-based adaptive gating mechanism. Extensive experiments on five benchmarks show that DSD is compatible with diverse teacher architectures, improves lightweight students across data regimes, and achieves a favorable accuracy–efficiency trade-off. Code is available at https://github.com/jyh0526/DSD.

Many real-world decisions depend on predicting future values from past records, such as electricity demand, traffic flow, weather conditions, or exchange rates. Powerful forecasting models can be accurate, but they are often too large and costly to run in practical settings, while smaller models are faster but may miss important patterns in the data. This paper studies how to make compact forecasting models more reliable without sacrificing their speed. We propose Dynamic Structural Distillation, a method that lets a small model learn from a larger model in a more careful way. Instead of forcing the small model to copy every forecast from the larger one, our method first checks whether the larger model’s guidance is likely to be trustworthy, especially when the data are noisy or changing over time. It also helps the small model learn more stable relationships hidden in the data, rather than only matching individual predicted values. We further design a lightweight forecasting model that can capture richer patterns while remaining efficient at inference time. Experiments on several real-world forecasting datasets show that our approach improves prediction accuracy while keeping computation low, making accurate forecasting more practical when resources are limited.