Timezone: »
Learned models and policies can generalize effectively when evaluated within the distribution of the training data, but can produce unpredictable and erroneous outputs on out-of-distribution inputs. In order to avoid distribution shift when deploying learning-based control algorithms, we seek a mechanism to constrain the agent to states and actions that resemble those that the method was trained on. In control theory, Lyapunov stability and control-invariant sets allow us to make guarantees about controllers that stabilize the system around specific states, while in machine learning, density models allow us to estimate the training data distribution. Can we combine these two concepts, producing learning-based control algorithms that constrain the system to in-distribution states using only in-distribution actions? In this paper, we propose to do this by combining concepts from Lyapunov stability and density estimation, introducing Lyapunov density models: a generalization of control Lyapunov functions and density models that provides guarantees about an agent's ability to stay in-distribution over its entire trajectory.
Author Information
Katie Kang (UC Berkeley)
Paula Gradu (UC Berkeley)
Jason Choi (UC Berkeley)
Michael Janner (UC Berkeley)
Claire Tomlin (UC Berkeley)
Sergey Levine (University of California, Berkeley)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Control »
Tue. Jul 19th through Wed the 20th Room Hall E #837
More from the Same Authors
-
2021 : Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention »
Abhishek Gupta · Justin Yu · Tony Z. Zhao · Vikash Kumar · Aaron Rovinsky · Kelvin Xu · Thomas Devlin · Sergey Levine -
2022 : You Only Live Once: Single-Life Reinforcement Learning via Learned Reward Shaping »
Annie Chen · Archit Sharma · Sergey Levine · Chelsea Finn -
2023 : Deep Neural Networks Extrapolate Cautiously in High Dimensions »
Katie Kang · Amrith Setlur · Claire Tomlin · Sergey Levine -
2023 : Deep Neural Networks Extrapolate Cautiously (Most of the Time) »
Katie Kang · Amrith Setlur · Claire Tomlin · Sergey Levine -
2023 : Data Diversity and Downstream impact »
Judy Hanwen Shen · Paula Gradu -
2023 : Safe Learning in Control »
Claire Tomlin -
2022 Social: Mental Health in ML Academia »
Paula Gradu · Cyril Zhang -
2022 Poster: Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning »
Philippe Hansen-Estruch · Amy Zhang · Ashvin Nair · Patrick Yin · Sergey Levine -
2022 Spotlight: Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning »
Philippe Hansen-Estruch · Amy Zhang · Ashvin Nair · Patrick Yin · Sergey Levine -
2022 Poster: Planning with Diffusion for Flexible Behavior Synthesis »
Michael Janner · Yilun Du · Josh Tenenbaum · Sergey Levine -
2022 Oral: Planning with Diffusion for Flexible Behavior Synthesis »
Michael Janner · Yilun Du · Josh Tenenbaum · Sergey Levine -
2022 Affinity Workshop: Women in Machine Learning (WiML) Un-Workshop »
Vinitra Swamy · Paula Gradu · Mojgan Saeidi · Noor Sajid · Shweta Khushu · Giulia Clerici · Tatjana Chavdarova -
2021 Poster: Offline Meta-Reinforcement Learning with Advantage Weighting »
Eric Mitchell · Rafael Rafailov · Xue Bin Peng · Sergey Levine · Chelsea Finn -
2021 Spotlight: Offline Meta-Reinforcement Learning with Advantage Weighting »
Eric Mitchell · Rafael Rafailov · Xue Bin Peng · Sergey Levine · Chelsea Finn