Curiosity-Bottleneck: Exploration By Distilling Task-Specific Novelty

Exploration based on state novelty has brought great success in challenging reinforcement learning problems with sparse rewards. However, existing novelty-based strategies become inefficient in real-world problems where observation contains not only task-dependent state novelty of our interest but also task-irrelevant information that should be ignored. We introduce an information-theoretic exploration strategy named Curiosity-Bottleneck that distills task-relevant information from observation. Based on the Information Bottleneck principle, our exploration bonus is quantified as the compressiveness of observation with respect to the learned representation of a compressive value network. With extensive experiments on static image classification, grid-world and three hard-exploration Atari games, we show that Curiosity-Bottleneck learns effective exploration strategy by robustly measuring the state novelty in distractive environment where state-of-the-art exploration methods often degenerate.