How2Everything: Mining the Web for How-to Procedures to Evaluate and Improve LLMs

Generating step-by-step "how-to" procedures is a key LLM capability: how-to advice is commonly requested in chatbots, and step-by-step planning is critical for complex reasoning tasks. Yet, measuring and improving procedural validity at scale on real-world tasks remains challenging and understudied. We introduce How2Everything, a scalable framework to evaluate and improve goal-conditioned procedure generation. Our pipeline How2Mine extracts and rewrites 351K procedures from 980K web pages across 14 topics, and can scale to larger corpora. From this pool we build How2Bench, a 7K-example evaluation set balanced across topics. We also introduce How2Score, an evaluation protocol that uses an LLM judge to detect whether a generation contains any critical failure that would prevent achieving the goal. For low-cost, reproducible evaluation, we distill a frontier judge into an open 8B model achieving 80.5\% agreement with human annotators. How2Bench reveals clear scaling trends across model size and training stages, providing signal early in pretraining. Finally, RL using How2Score as a reward improves performance on How2Bench by >10 points across three base models without systematic regressions on standard benchmarks, with gains not primarily explained by source-document memorization or superficial format compliance. We release all code and data at https://github.com/lilakk/how2everything.

People often turn to AI assistants for step-by-step help with a variety of tasks, like cooking a recipe, fixing things around the house, or filling out legal forms. For many of these tasks, even when the answers sound plausibly useful, they can miss an important step or include a small mistake that causes the whole procedure to fail. We introduce How2Everything, a framework for evaluating and improving how well AI systems provide how-to advice. We collect a wide range of real procedures from the web and turn them into a broad test set. We also build an automatic judge that identifies serious failures in the steps generated by AI models. Our results show that stronger models perform better on the benchmark, and that training models with feedback from our judge leads to better performance without hurting other useful AI capabilities like coding and mathematical reasoning. Overall, we offer a practical way to evaluate AI-generated step-by-step advice and make it more reliable.