SWE-IF: Aligning Code Evaluation with Human Preference

Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models' code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.

More and more people ask AI to write code, often just by describing what they want in plain words. But code that runs is not always good code. It should also do what you actually asked for, like following a clean style or staying easy to read. The trouble is that today's tests for AI coding tools check only whether the code gives the right answer, and quietly ignore whether it follows the rest of your instructions. To study this blind spot, we write down 30 clear coding instructions that a computer can check automatically, and use them to build a new test. The test scores AI-written code on two things at once. Does it work, and does it do what it was told? We then run this test on 31 of today's top AI models. Even the strongest models struggle. The more instructions you give them, the worse they get at following them, and the more often they break the code. We also find that judging code on both things together matches what people actually prefer far better than checking correctness alone. So if we want AI coding assistants that people enjoy using, we should test and train them on following instructions, not just on getting the right answer.