Time-Consistent Robust Multi-Objective Reinforcement Learning via a Bellman–Isaacs Weight-Adversary Recursion

Most multi-objective reinforcement learning (MORL) methods either condition on a fixed preference weight $w$ or consider episodic robustness where an adversary selects a single $w$ per episode. We study a time-consistent robustness model with reactive preferences: after each transition, an opponent chooses the next weight $w_{t+1}$ after observing $s_{t+1}$, and incurs a switching cost $\lambda D_\Phi(w_{t+1}\mid w_t)$ based on a Bregman divergence. This yields a Bellman–Isaacs recursion with an inner weight minimization at every backup. We prove the induced operator is a contraction and derive a Bellman-residual certificate that turns approximation error into a uniform bound on robust performance. We develop practical solvers in both tabular and deep settings using Bregman-prox inner updates and a stabilized fixed-point iteration. To evaluate robustness without optimistic critic reuse, we introduce BR-$K$, testing policies against $K$ independently trained best-response preference adversaries. Across MO-Gymnasium benchmarks, our approach consistently improves WRR under strong step-wise opponents over preference-conditioned baselines while keeping DRIFT smoothly controllable via $\lambda$.

Many reinforcement learning systems must balance several goals at once, such as speed, safety, cost, and reward. Existing methods usually assume that the user's preference over these goals is fixed during an episode. In practice, however, preferences may change as the situation changes. For example, after an agent reaches a risky state, safety may suddenly become more important than speed. This paper studies reinforcement learning under such changing preferences. We model the preference as something that can be adjusted step by step by an adversary, while also charging a cost for changing preferences too abruptly. This creates a more demanding test of whether a policy remains reliable when priorities shift during deployment. We develop algorithms that learn policies for this setting in both small tabular problems and larger problems using neural networks. We also introduce an evaluation protocol that tests a learned policy against independently trained adversaries, rather than relying only on the model used during training. Experiments on multi-objective reinforcement learning benchmarks show that our method is more robust to step-by-step preference changes than standard preference-conditioned baselines.