The field of reinforcement learning is pushed forwards by the presence of challenging environments. Over the years, the complexity of these environments has continued to increase, but the question is how can we continue to push the complexity of environments with respect to the optimal policy complexity in a scalable manner. Here I will discuss using multi-agent environments to create more open-ended environments, and discuss examples of our work to move in this direction with Capture the Flag and Starcraft 2. Finally I will discuss some future directions for generating even more open-ended environments to further push our RL algorithms.