HE-SNR: Uncovering Latent Logic via Entropy for Guiding Mid-Training on SWE-bench

SWE-bench has emerged as the premier benchmark for evaluating Large Language Models on complex software engineering tasks. While these capabilities are fundamentally acquired during the mid-training phase and subsequently elicited during Supervised Fine-Tuning (SFT), there remains a critical deficit in metrics capable of guiding mid-training effectively. Standard metrics such as Perplexity (PPL) are compromised by the "Long-Context Tax" and exhibit weak correlation with downstream SWE performance. In this paper, we bridge this gap by first introducing a rigorous data filtering strategy. Crucially, we propose the Entropy Compression Hypothesis, redefining intelligence not by scalar Top-1 compression, but by the capacity to structure uncertainty into Entropy-Compressed States of low orders ("reasonable hesitation"). Grounded in this fine-grained entropy analysis, we formulate a novel metric, HE-SNR (High-Entropy Signal-to-Noise Ratio). We validate our approach on models with up to 560B parameters across different context windows (32K/128K). This work provides both the theoretical foundation and practical tools for optimizing the latent potential of LLMs in complex engineering domains.

Modern AI models can generate and reason about computer code, but it is difficult to measure which parts of their decision-making actually lead to successful solutions. In this work, we introduce SWE-bench, a benchmark for evaluating AI on complex software engineering tasks, and HE-SNR, a new method to track and guide models during training. HE-SNR identifies the most informative steps in a model’s reasoning and filters out irrelevant or misleading signals from the training process. By focusing on these high-value steps, our approach helps AI models learn more effectively, improving their ability to solve challenging coding and reasoning tasks. Tested on models with up to 560 billion parameters, this method provides practical tools for building AI assistants that can understand, write, and debug code, potentially enhancing software development workflows.