PACE: Parameter Change for Unsupervised Environment Design

Unsupervised Environment Design (UED) offers a promising paradigm for improving reinforcement learning generalization by adaptively shaping training environments, but it requires reliable environment evaluation to remain effective. However, existing UED methods evaluate environments using indirect proxy signals such as regret, value-based errors, or Monte Carlo, which suffer from bias, high variance, or substantial computational overhead. To address these limitations, we propose Parameter Change Environment Design (PACE), a general framework for adaptive level selection in UED. PACE evaluates a level by performing a provisional policy update on it and scoring it with the squared $\ell_2$ norm of the induced parameter change, which directly reflects realized learning progress. This score then guides level selection: levels enter a staleness-aware buffer based on their score, and are replayed via rank-based prioritization, inducing a curriculum that adapts to the agent’s evolving capability. By grounding environment evaluation in intrinsic optimization progress, PACE provides a low-variance evaluation signal and avoids the need for additional environment rollouts. Experiments on MiniGrid and Craftax demonstrate that PACE consistently outperforms established UED baselines in zero-shot generalization across diverse out-of-distribution evaluation protocols.

When training AI agents to solve tasks — like navigating a maze or surviving in a virtual world — it matters a great deal which practice scenarios they train on. If the training environments are too easy or too repetitive, the agent fails to generalize when faced with new, unseen challenges. Existing methods for automatically selecting good training environments rely on indirect and noisy signals, making them slow, inconsistent, or computationally expensive. We propose PACE, a framework that selects training environments by directly measuring how much an agent actually learns from each one. Concretely, after a trial run on a candidate environment, we measure how much the agent's internal parameters would shift — a simple, reliable indicator of genuine learning progress. Environments that drive meaningful change are prioritized; those the agent has already mastered are set aside. This creates a self-adjusting curriculum that keeps pace with the agent's growing abilities. PACE is simpler and more efficient than prior approaches, yet consistently produces agents that generalize better to new, unseen environments — a critical property for deploying AI systems in the real world.