Hybrid Reinforcement Learning in Adversarial Markov Decision Processes

We study hybrid reinforcement learning (RL) in adversarial Markov Decision Processes (MDPs), where the learner simultaneously receives on-policy feedback from the executed policy and off-policy feedback from a fixed behavior policy, and loss functions can change arbitrarily over time. On-policy feedback allows exploration and ensures the worst-case guarantee against any comparator policy, while off-policy feedback provides coverage-dependent guarantee that scales with the "mismatch" between the behavior and comparator policies (called coverage ratio) and can be sharper than on-policy results whenever this ratio is small. We propose a new hybrid RL framework that accommodates adversarial losses and unknown transitions, preserving off-policy guarantees while ensuring non-trivial worst-case performance.

Imagine training an AI system that can learn from its own choices, while also watching an older system that is already being used. If the older system often makes useful choices, watching it can speed up learning. But if it misses important choices, relying on it too much can hurt. This paper develops a way to combine these two kinds of feedback. The learner uses the older system when it is helpful, but can fall back on its own exploration when needed. Our results show that this combined approach can learn faster in favorable cases while still protecting against bad cases. This is useful for building AI systems that must keep learning after deployment, especially when older data or existing policies are available but imperfect.