Retrospective and Structurally Informed Exploration via Cross-task Successor Feature Similarity

Intrinsically motivated exploration in reinforcement learning typically relies on novelty, prediction error, or surprise to guide agents toward underexplored states. However, these signals often ignore valuable structural knowledge gained from prior tasks, leading to inefficient or redundant exploration. We introduce Cross-task Successor Feature Similarity Exploration (C-SFSE), a novel intrinsic reward mechanism that leverages retrospective similarities in task-conditioned successor features to prioritize exploration of semantically meaningful states. C-SFSE constructs a cross-task similarity signal from previously learned policies, identifying regions, such as bottlenecks or reusable subgoals, that consistently support goal-directed behavior. This enables the agent to focus its exploration on state space areas that are not only novel but informative across tasks. We evaluate C-SFSE in continuous control tasks to demonstrate its effectiveness in realistic and challenging settings where traditional count-based or discrete exploration methods often fall short. Specifically, we show that C-SFSE enables structured, sample-efficient exploration in high-dimensional action spaces, as evidenced by its performance across several MuJoCo environments. Our experiments demonstrate that C-SFSE consistently outperforms existing intrinsic motivation and successor feature-based exploration approaches in terms of both sample efficiency and overall performance.