Abstract: In Inverse Reinforcement Learning (IRL), we learn the underlying reward function of humans from observations. Recent work shows that we can learn the reward function more accurately by observing the human in multiple related environments, but efficiently finding informative environments is an open question. We present $\texttt{AMBER}$, an information-theoretic algorithm that generates highly informative environments. With theoretical and empirical analysis, we show that $\texttt{AMBER}$ efficiently finds informative environments and improves reward learning.