REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillation

Diffusion models have significantly advanced the field of talking head generation (THG). However, slow inference speeds and prevalent non-autoregressive paradigms severely constrain the application of diffusion-based THG models. In this study, we propose REST, a pioneering diffusion-based, real-time, end-to-end streaming audio-driven talking head generation framework. To support real-time end-to-end generation, a compact video latent space is first learned through a spatiotemporal variational autoencoder with a high compression ratio. Additionally, to enable semi-autoregressive streaming within the compact video latent space, we introduce an ID-Context Cache mechanism, which integrates ID-Sink and Context-Cache principles into key-value caching for maintaining identity consistency and temporal coherence during long-term streaming generation. Furthermore, an Asynchronous Streaming Distillation (ASD) strategy is proposed to mitigate error accumulation and enhance temporal consistency in streaming generation, leveraging a non-streaming teacher with an asynchronous noise schedule to supervise the streaming student. REST bridges the gap between autoregressive and diffusion-based approaches, achieving a breakthrough in efficiency for applications requiring real-time THG. Experimental results demonstrate that REST outperforms state-of-the-art methods in both generation speed and overall performance.

Talking-head generation aims to create a video of a person speaking from an audio recording, which is useful for live avatars, online meetings, virtual assistants, and remote education. Although recent methods can produce lifelike results, they are often too slow for live conversations, causing noticeable delays before the video appears. To solve this, we developed REST, a new framework that generates talking-head videos efficiently in real-time. Instead of working directly with large video frames, REST first learns a much smaller internal description of the video, allowing the framework to work much faster. We also designed a smart memory feature that selectively holds onto key details from earlier in the video. This ensures the speaker continues to look like themselves and moves naturally, even during long conversations. To make the generation results more reliable, we trained our fast, real-time framework with guidance from a stronger offline model, which helps reduce errors that can accumulate over time and enhance performance. Our tests show that REST is not only faster but also produces more convincing videos than existing methods, making high-quality digital humans practical for everyday, interactive applications.