Why Dedicated Critics: Eliminating Target Drift in Multi-Constraint RL

Lagrangian-based methodologies are one of the fundamental paradigms of safe reinforcement learning (RL) for constrained Markov decision processes, particularly when dealing with multi-constraint cases. While the specific details of the methodologies may differ, with some using a single estimator for the overall mixed penalty term of the constraints and others using separate estimators for the constraints, the fundamental question of the theoretical validity of the methodologies has remained largely unexplored. The present paper performs the first theoretical analysis of the methodologies and proves that the use of the mixed critic structure leads to the presence of a bias due to the target drift of the Lagrange multipliers. On the other hand, the use of the dedicated critic structure, where separate critics are used for the reward function and the constraint functions, does not suffer from this bias. The theoretical analysis is supported with experiments on a realistic power system environment with multiple constraints, where the dedicated critic structure succeeds in satisfying the constraints, whereas the mixed critic structure fails.

Many AI systems learn by trying actions and receiving feedback. In safety-critical areas such as power grids, robotics, and transport, it is not enough for such a system to perform well; it must also obey several safety and resource rules at the same time. Existing methods often combine all these rules into one overall safety score while the system is learning. This paper shows that this common design can be unreliable: as the importance of different safety rules changes during training, the combined score also keeps changing, which can confuse the learning process and lead to more safety violations. We study an alternative design that keeps separate internal evaluators for the main goal and for each safety rule. This makes each evaluator’s job clearer and prevents the learning target from shifting unnecessarily. We provide a mathematical explanation for why this design is more stable, and we test it in both a simple example and realistic power-system control settings with multiple operational constraints. The results show that the separate-evaluator design learns more stably and satisfies safety constraints more reliably. This provides practical guidance for building safer learning-based control systems in real-world applications.