Return of Frustratingly Easy Unsupervised Video Domain Adaptation

Unsupervised video domain adaptation (UVDA) is a practical but under-explored problem. In this paper, we propose a frustratingly easy UVDA method, called \emph{MetaTrans}. Specifically, \emph{MetaTrans} adopts a concise learning objective that contains only two fundamental loss terms. Despite the simplicity of the learning objective, \emph{MetaTrans} embodies an advanced UVDA idea, that is, handling the spatial and temporal divergence of cross-domain videos separately, through a subtle model architecture design. By implementing a temporal-static subtraction module, \emph{MetaTrans} effectively removes spatial and temporal divergence. Extensive empirical evaluations, particularly on various cross-domain action recognition tasks, show substantial absolute adaptation performance enhancement and significantly superior relative performance gain compared with state-of-the-art UVDA baselines.

Videos collected in different places often look and move differently: a model trained to recognize actions in one dataset may struggle in another, and labeling many new videos is costly. This paper studies how to adapt action-recognition models to a new video collection without using labels from that collection. The proposed method, MetaTrans, separates two sources of mismatch: static appearance, such as background, camera style, or scene, and motion over time. It uses a simple two-stream transformer design: one stream learns time-aware video information, while the other estimates order-independent static information; subtracting the latter helps focus learning on the action itself. With only two training losses, MetaTrans is much simpler to tune than many previous approaches. Experiments on standard video adaptation benchmarks show that this simplicity does not sacrifice performance, achieving strong accuracy and better performance gain per training attempt. This could reduce labeling and tuning costs for real-world video understanding systems.