Paper #72: Forty Shades of Blue: Quality-Diversity Alignment \\ via Mode-Conditioned Reinforcement Learning

A notable byproduct of LLM alignment training is mode collapse: a progressive loss of output diversity that narrows the model’s expressive capacity at inference time. This degradation is especially limiting for applications requiring creativity, open-ended exploration, and pluralistic perspectives. We present MODA (MOde-conditioned Diversity Alignment), a post-training RL algorithm that jointly optimizes generation quality and diversity. MODA trains a single shared policy over abstract numbered mode tokens, enabling distinct mode-conditioned behaviors that explore different regions of the high-quality output space without hand-crafted personas or architectural changes. We further introduce a Quality-Gated Diversity Reward, which rewards diversity only when outputs exceed a prompt-adaptive quality threshold calibrated against a frozen reference policy. This gating mechanism prevents degenerate reward-hacking behaviors such as language switching, verbosity, and irrelevant responses, while preserving generation quality. To study quality-diversity tradeoffs, we evaluate MODA on a comprehensive benchmark suite comprising five general capability tasks and four domain-specific diversity tasks spanning scientific ideation and creative writing. MODA improves SBERT diversity by 29.1% while simultaneously increasing general capability performance by 22.4% over the base model. MODA offers a drop-in alternative to standard post-training methods that otherwise contract the model’s exploration space.