Better, Faster: Harnessing Self-Improvement in Large Reasoning Models

Self-improvement training enables the large reasoning models (LRMs) to improve themselves by self-generating reasoning trajectories as training data without external supervision. However, we find that this method often falls short in complex reasoning tasks and even leads to model collapse. Through a series of preliminary analyses, we reveal two problems: (1) data imbalance, where most training samples are simple, but the challenging yet crucial samples are scarce; (2) overthinking, where many undesired samples with redundant reasoning steps are used for self-training. To this end, we propose HSIR, which effectively Harnesses Self-Improvement in large Reasoning models via two simple-yet-effective approaches. Specifically, HSIR introduces a verify-then-exit sampling strategy to mitigate data imbalance by efficiently collecting more accurate solutions for difficult queries, and designs an Intrinsic Diversity score to quantify overthinking and filter out the undesired solutions. We apply HSIR to various post-training paradigms, among which we further propose H-GRPO, an enhanced GRPO algorithm that leverages the intrinsic diversity as an external reward to encourage concise and diverse reasoning via reinforcement learning. Extensive results show that HSIR not only effectively enhances the reasoning performance, i.e., bringing up to +10.9% average performance gains, but also significantly improves the reasoning efficiency by reducing up to 42.4% relative inference overhead.

Large reasoning models solve complex problems by writing out their thinking step by step before giving a final answer. A promising way to improve these models is self-improvement: letting them practice on problems and learn from their own correct solutions. However, we found this approach often fails on hard tasks for two reasons: the model mostly practices easy problems while neglecting difficult ones, and it develops a habit of overthinking — repeating similar reasoning steps in circles without making progress. We propose HSIR, a framework that fixes both issues. Instead of throwing away failed attempts, we salvage the useful parts where the model was on the right track before going astray, turning them into new correct examples for harder problems. To tackle overthinking, we designed a method that looks inside the model's internal representations to detect and filter out repetitive, redundant reasoning. Across seven models and five different tasks, HSIR improves reasoning accuracy by up to 10.9% while cutting unnecessary computation by up to 42.4%, making these models both smarter and faster.