PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching

Unsupervised Reinforcement Learning from Internal Feedback (RLIF) has emerged as a promising paradigm for eliciting the latent capabilities of Large Language Models (LLMs) without external supervision. However, current methods rely on heuristic intrinsic rewards, which often lack a well-defined theoretical optimization target and are prone to degenerative biases. In this work, we introduce PowerFlow, a principled framework that reformulates unsupervised fine-tuning as a distribution matching problem. By casting GFlowNet as an amortized variational sampler for unnormalized densities, we propose a length-aware Trajectory-Balance objective that explicitly neutralizes the structural length biases inherent in autoregressive generation. By targeting $\alpha$-power distributions, PowerFlow enables the directional elicitation of the dual nature of LLMs: sharpening the distribution ($\alpha > 1$) to intensify logical reasoning, or flattening it ($\alpha < 1$) to unlock expressive creativity. Extensive experiments demonstrate that PowerFlow consistently outperforms existing RLIF methods, matching or even exceeding supervised GRPO. Furthermore, by mitigating over-sharpening in aligned models, our approach achieves simultaneous gains in diversity and quality, shifting the Pareto frontier in creative tasks.

Modern AI chatbots have hidden abilities they don't always show. They can sometimes reason better than they appear to, yet the same training that makes them helpful also tends to make their writing bland and predictable. Methods that try to coax out these skills without human-labeled answers rely on hand-crafted tricks that often backfire — models grow overconfident, give absurdly short replies, or collapse to a single style. We introduce PowerFlow, a principled framework that reshapes a model's own probability distribution instead of chasing hand-designed rewards. A single tunable knob controls the direction: turning it up concentrates the model on its most promising answers, sharpening logical reasoning; turning it down spreads attention across rarer alternatives, restoring creativity. Because language models write one word at a time, naive versions of this idea let the model "cheat" by producing short or repetitive text, so we designed a length-aware training objective that closes this loophole. Despite using no labeled training data, PowerFlow matches or surpasses methods that rely on human-labeled answers on math and graduate-level science problems; on creative-writing tasks, it simultaneously raises both diversity and quality — giving researchers one principled dial that tells an AI whether to think harder or write more freely.