Timezone: »

A Theory of Regularized Markov Decision Processes
Matthieu Geist · Bruno Scherrer · Olivier Pietquin

Tue Jun 11 06:30 PM -- 09:00 PM (PDT) @ Pacific Ballroom #116

Many recent successful (deep) reinforcement learning algorithms make use of regularization, generally based on entropy or Kullback-Leibler divergence. We propose a general theory of regularized Markov Decision Processes that generalizes these approaches in two directions: we consider a larger class of regularizers, and we consider the general modified policy iteration approach, encompassing both policy iteration and value iteration. The core building blocks of this theory are a notion of regularized Bellman operator and the Legendre-Fenchel transform, a classical tool of convex optimization. This approach allows for error propagation analyses of general algorithmic schemes of which (possibly variants of) classical algorithms such as Trust Region Policy Optimization, Soft Q-learning, Stochastic Actor Critic or Dynamic Policy Programming are special cases. This also draws connections to proximal convex optimization, especially to Mirror Descent.

Author Information

Matthieu Geist (Google)
Bruno Scherrer (INRIA)
Olivier Pietquin (GOOGLE BRAIN)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors