Knothe-Rosenblatt Quantile Regression for Risk-sensitive Multi-objective Reinforcement Learning

In this work, we extend distributional reinforcement learning (RL) to develop a risk-sensitive multi-objective RL framework, with applications to domains such as finance and robotics. We achieve this by adopting vector-risk measures and approximating them via Knothe-Rosenblatt (KR) quantile regression. This approach directly extends the IQN framework to the multi-objective setting, aligns with the axiomatic definition of vector-risk measures, and guarantees that critics converge under the distributional Bellman operator. To mitigate the artificial ordering imposed by the KR map, we employ a transformer architecture without positional encoding, and introduce MO-TQC for training stability. We demonstrate improved performance on MO-Gymnasium benchmarks and use our framework to study risk-sensitive policies in multi-objective tasks.

This paper addresses sequential decision-making problems (Reinforcement Learning, RL) involving multiple objectives, with a specific focus on risk-sensitive scenarios. We are particularly interested in risk-sensitive situations where careful attention to these objectives is required. To handle this, we utilize a value-by-value (auto-regressive) evaluation method, which means we evaluate our progress sequentially, with each step building upon the outcomes of the previous ones. Note that this sequence does not depend on the decision-making time steps, but rather on a predefined order of the objectives (e.g., evaluating the robot's velocity first, and then its energy consumption). We also provide the mathematical justification of this approach and introduce practical techniques to prevent the algorithm from being overly optimistic about its decisions.