CE$^4$L: Continual Ego, Exo, and Ego-Exo Learning

Perception for embodied agents is video-based, often multi-view (ego, exo, or both), and inherently continual, with simultaneous task and viewpoint shifts. Yet continual learning (CL) remains dominated by exo-only recognition tasks, obscuring behavior under these real-world coupled shifts. We introduce **C**ontinual **E**go, **E**xo, and **E**go-**E**xo **L**earning (**CE$^4$L**), a unified multi-view CL benchmark spanning four representative tasks: cross-view referenced skill assessment, temporal action segmentation, cross-view association, and action anticipation \& planning. CE$^4$L highlights challenges largely absent in prior CL benchmarks, including cross-view correspondence, view-dependent asynchrony, and heterogeneous semantic objectives. To this end, we propose **V**ideo **I**ncremental **S**ubspace-routed **T**ask **A**dapters (**VISTA**), a parameter-efficient baseline method that stores task-specific updates in lightweight adapters and performs training-free routing via residual distance to task-specific whitened subspaces estimated from second-order statistics. Extensive experiments demonstrate the significantly varied efficacy of representative CL methods across CE$^4$L settings, while VISTA is consistently competitive and achieves state-of-the-art overall performance.

AI agents such as robots often learn from videos while their tasks and camera viewpoints change over time. For example, the same activity can look very different from a first-person camera and from an external camera, and a model that remembers old tasks may still fail to transfer knowledge across these views. We introduce CE4L, a benchmark that tests continual learning in multi-view video settings, including skill assessment, action segmentation, cross-view matching, and action anticipation and planning. This benchmark helps researchers measure not only whether a model forgets past tasks, but also whether it preserves knowledge that transfers between egocentric and exocentric views. We also propose VISTA, a memory-efficient method that stores task-specific adaptations and automatically selects the most relevant ones using statistics of video representations. Across CE4L, VISTA performs strongly compared with representative continual learning methods. We hope CE4L will help the community build video learning systems that remain stable and transferable in realistic embodied environments.