Boosting CVaR Policy Optimization with Quantile Gradients

Optimizing Conditional Value-at-risk (CVaR) using policy gradient (a.k.a CVaR-PG) faces significant challenges of sample inefficiency. This inefficiency stems from the fact that it focuses on tail-end performance and overlooks many sampled trajectories. We address this problem by augmenting CVaR with an expected quantile term. Quantile optimization admits a dynamic programming formulation that leverages all sampled data, thus improves sample efficiency. This does not alter the CVaR objective since CVaR corresponds to the expectation of quantile over the tail. Empirical results in domains with verifiable risk-averse behavior show that our algorithm within the Markovian policy class substantially improves upon CVaR-PG and consistently outperforms other existing methods.

Standard reinforcement learning methods usually focus on average performance, but in safety-critical situations, we must prepare for worst-case scenarios. To prevent rare but catastrophic failures, researchers use a risk measure called Conditional Value-at-Risk (CVaR) to optimize the worst outcomes. However, current algorithms are highly inefficient because they throw away most of the data and only look at the worst-performing attempts. We solved this problem by introducing a mathematical shortcut that allows the algorithm to learn from all collected experiences, not just the bad ones, without changing the final safety goals. By using this data-leveraging approach, our method trains risk-aware policy much faster and more reliably. Testing shows our algorithm consistently outperforms existing methods in environments where avoiding risk is absolutely vital.