Timezone: »

 
Oral
Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control
Yangchen Pan · Amir-massoud Farahmand · Martha White · Saleh Nabi · Piyush Grover · Daniel Nikovski

Fri Jul 13 08:20 AM -- 08:40 AM (PDT) @ A1
in Reinforcement Learning 17 »

Recent work has shown that reinforcement learning (RL) is a promising approach to control dynamical systems described by partial differential equations (PDE). This paper shows how to use RL to tackle more general PDE control problems that have continuous high-dimensional action spaces with spatial relationship among action dimensions. In particular, we propose the concept of action descriptors, which encode regularities among spatially-extended action dimensions and enable the agent to control high-dimensional action PDEs. We provide theoretical evidence suggesting that this approach can be more sample efficient compared to a conventional approach that treats each action dimension separately and does not explicitly exploit the spatial regularity of the action space. The action descriptor approach is then used within the deep deterministic policy gradient algorithm. Experiments on two PDE control problems, with up to 256-dimensional continuous actions, show the advantage of the proposed approach over the conventional one.

[ PDF] 

Author Information

Yangchen Pan (University of Alberta)
Amir-massoud Farahmand (Vector Institute)
Martha White (University of Alberta)
Saleh Nabi
Piyush Grover (Mitsubishi Electric Research Labs)

Piyush Grover is a principal researcher at MERL. He obtained his Ph.D. in Engineering Mechanics in 2010 from Virginia Tech, under the supervision of Shane Ross. His work involves a mix of basic and applied research at the intersection of nonlinear dynamical systems, mechanics and control. He is interested in both geometric/topological and operator-theoretic (or statistical) descriptions of phase space transport in dynamical systems, and deriving low-order descriptions of distributed systems.

Daniel Nikovski (Mitsubishi Electric Research Labs)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors