Lifelong reinforcement learning holds promise to enable autonomous decision making in applications ranging from household robotics to autonomous space exploration. In this talk we will discuss key challenges that need to be address to make progress towards this grand vision. First, we discuss the fundamental need for the community to invest in a shared understanding of problem formalizations and evaluation paradigms and benchmarks. Second, we dive deeper into ongoing work towards tackling exploration and representation in lifelong reinforcement learning. Our aim is to spark debate and inspire research in this exciting space.