HTAC: Hierarchical Task-Aware Composition for Continual Offline Reinforcement Learning

Continual Offline Reinforcement Learning (CORL) enables building long-term autonomous agents from static datasets. However, it faces heterogeneity in environment dynamics, reward functions, and behavior policies across tasks. Combined with the inherent distribution shift in offline learning, this requires agents to selectively reuse shared knowledge during transfer while isolating task-specific features. The flat knowledge sharing mechanisms employed by existing methods struggle to capture such distinctions, limiting cross-task generalization. To address this, we propose Hierarchical Task-Aware Composition (HTAC), which balances plasticity and stability through dual-level task encoding and soft composition mechanisms. HTAC comprises four modules: (1) a Hierarchical Semantic Task Representation that decomposes tasks into domain-level and task-level embeddings; (2) a Dual-level Expert Network that creates domain and task experts on demand for parameter-efficient knowledge isolation; (3) an Adaptive Knowledge Composition module that integrates historical expert outputs via attention mechanisms for knowledge reuse; (4) Task Adapters that preserve historical routing weights to prevent forgetting. Experiments on Offline Continual World show that HTAC outperforms existing baselines, demonstrating better knowledge reuse and transfer capabilities.

Autonomous agents such as robots often need to learn new skills over time — for example, first learning to push an object, then to grasp, then to place — using only pre-recorded demonstration data, without any live trial-and-error. A key difficulty is that learning a new skill can cause the agent to forget previously learned ones, a problem known as catastrophic forgetting. Existing approaches try to address this by either constraining how the model updates or by replaying old data, but they treat all knowledge uniformly and fail to distinguish what is shared across tasks from what is unique to each task, limiting both knowledge reuse and retention. We propose HTAC (Hierarchical Task-Aware Composition), which explicitly separates knowledge into two levels: domain-level knowledge shared among related tasks, and task-level knowledge specific to each individual task. HTAC selectively composes relevant historical knowledge through a semantic similarity-based attention mechanism, and creates new expert modules only when existing knowledge is insufficient. Per-task adapters protect previously learned routing patterns from being overwritten. On the Offline Continual World benchmark, HTAC outperforms existing methods, showing stronger knowledge reuse, better transfer to later tasks, and less forgetting of earlier ones.