Distill, Suppress, and Fuse: Cross-Modal Knowledge Integration for Optical Flow-Free Temporal Action Segmentation

Cross-modal knowledge distillation (CMKD) enables efficient inference by transferring knowledge from a teacher model trained on a computationally heavy modality (i.e., optical flow) to a student model operating on a lightweight modality (i.e., RGB). However, we find that most current CMKD methods are hindered by a key limitation when applied in temporal action segmentation: motion cues transferred from optical flow often lead the student to produce frame representations that are misaligned with the underlying action structure. To address this, we propose RELATE, an optical flow-free framework that selectively integrates transferred cues while suppressing misaligned cues. We further introduce a prediction refinement strategy to resolve ambiguous segments using multiple predictions. Experiments on three benchmarks with multiple segmenters show that RELATE consistently outperforms RGB-only baselines, approaches two-stream performance, and achieves up to 175× faster inference.