RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization

In safety-critical domains where online data collection is infeasible, offline reinforcement learning (RL) is attractive only if policies achieve high returns without catastrophic lower-tail risk. Prior work on risk-averse offline RL achieves safety at the cost of value- or model-based pessimism, and restricted policy classes that limit policy expressiveness, whereas diffusion/flow-based expressive generative policies have largely been used in risk-neutral settings. We introduce **Risk-Aware Multimodal Actor-Critic (RAMAC)**, a simple, modular, model-free framework that couples an expressive generative actor (e.g., diffusion/flow) with a distributional critic and optimizes a composite objective that combines Conditional Value-at-Risk (CVaR) with behavioral cloning (BC), enabling risk-sensitive learning in complex multimodal scenarios. Since out-of-distribution (OOD) actions are a major driver of catastrophic failures in offline RL, we further provide an objective-level analysis showing that controlling behavior divergence via BC suppresses OOD actions and stabilizes CVaR. Instantiating RAMAC with a diffusion actor, we illustrate these insights on a 2-D risky bandit and evaluate on Stochastic-D4RL, observing consistent gains in CVaR$_{0.1}$ while maintaining strong returns.

Imagine a robot or autonomous system that usually performs well, but occasionally makes a catastrophic mistake. When collecting new trial-and-error data is costly or unsafe, we would like it to learn from previously collected experience while avoiding such rare failures. Recent generative models can represent several valid ways of acting, which is valuable for complex control tasks, but methods that focus on high average performance may still overlook dangerous worst-case outcomes. We introduce RAMAC, a method that combines flexible action generation with explicit attention to the worst-performing outcomes. RAMAC learns a range of possible consequences for each action, then favors actions whose worst cases are safer while keeping the policy close to behaviors supported by the offline data. On simulated locomotion tasks with rare severe penalties, our diffusion-based implementation improved performance in the worst 10% of trials while preserving strong average performance and producing fewer unsupported actions than representative risk-aware alternatives. RAMAC is a step toward learning-based controllers that remain capable and flexible without ignoring rare but severe failures.