Refined Analysis of Entropy-Regularized Actor-Critic

In this paper, we study the role of the critic in actor-critic for entropy-regularized, finite, discounted environments. We establish that, when the critic is exact, using the latter as a baseline is an actual variance-reduction method. In this case, actor-critic with stochastic gradients matches the sample complexity of deterministic policy gradient, reaching an $\epsilon$-optimal regularized value with $\tilde{O}(\log(1/\epsilon))$ samples. In practice, the critic is learned alongside the actor: the variance of the actor update is then influenced by the critic's variance and bias. Specifically, when the critic has a sufficiently small error, the variance reduction and rapid convergence are preserved. This suggests to learn the critic first, keeping it up to date after each actor update, underscoring the pivotal role of accurate critic estimation in actor-critic methods.

Reinforcement learning is a way of training artificial intelligence systems to make decisions by trial and error. Many successful reinforcement learning methods use two components: one component, called the actor, chooses actions, while another component, called the critic, evaluates how good these actions are. Although actor–critic methods are widely used in practice, it is still not fully understood why the critic helps training so much. In this paper, we study this question mathematically in a simplified but important setting. We show that when the critic is accurate, it can greatly reduce the randomness in the actor’s learning updates. In fact, the randomness disappears as the algorithm approaches the best solution, which explains why actor–critic methods can converge much faster than standard policy-gradient methods. We also study the realistic case where the critic must be learned during training. Our results show that the overall performance then depends strongly on how accurately the critic is estimated. In particular, spending more computation on improving the critic can lead to better and more stable learning. Our experiments support this conclusion: increasing the number of critic updates consistently improves the performance of the learned policy. Overall, the paper provides a clearer theoretical explanation for the central role of the critic in actor–critic reinforcement learning methods.