EigenCache: Rethinking Diffusion Acceleration as Covariance-Optimal Forecasting and Submodular Information Allocation

Accelerating diffusion models via feature caching has progressed from static feature reuse to polynomial extrapolation, yet current cache-then-forecast strategies still rely on hand-crafted approximation families (e.g., Taylor or Hermite bases) that can misalign with the non-stationary, layer-specific dynamics of generative features. This paper introduces EigenCache, a training-free framework that re-frames diffusion acceleration as covariance-adaptive feature forecasting and uncertainty-aware temporal design. EigenCache models cached feature trajectories as time-indexed stochastic processes and estimates layer-wise temporal kernels from a small calibration set. Under the resulting scalar temporal-kernel approximation, the Gaussian-process posterior mean, i.e., Kriging, is risk-optimal within the scalar-weighted linear predictor class; under joint Gaussianity, it further coincides with the MMSE estimator. This formulation generalizes fixed-basis forecasting from a covariance-adaptive perspective while providing a closed-form posterior-variance proxy for prediction uncertainty. Leveraging this proxy, EigenCache selects computation anchors by maximizing a log-determinant information-gain objective over denoising timesteps, a monotone submodular objective with a near-optimal greedy solution. Across image, video, transformer, U-Net, and LoRA-adapted diffusion models, EigenCache achieves a strong speed--fidelity Pareto frontier and provides a principled mechanism for robust compute allocation.

High-quality image and video generation with modern AI is often slow and computationally expensive, because these models repeatedly compute intermediate features at many steps. We observed that many of these features follow predictable temporal patterns, suggesting an opportunity to accelerate the process. We present EigenCache, a principled framework that learns the temporal relationships of these features from a small calibration set, allowing the model to “forecast” uncomputed features while selectively computing the most informative ones. Unlike prior methods that rely on rigid polynomial approximations, EigenCache adapts to the true dynamics of the generative process, providing uncertainty-aware scheduling that ensures high fidelity. Experiments show that it consistently achieves faster generation across images, videos, multiple architectures, and models augmented with adapters, without sacrificing visual quality. This approach reduces energy usage, enables near real-time content creation, and provides a general mechanism for efficient generative AI, making high-fidelity synthesis more accessible and sustainable.