CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

LLM coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refactoring: behavior-preserving program transformations that improve structure and maintainability. We investigate whether agents (i) can execute refactorings reliably and (ii) identify the refactorings that human developers actually chose in real codebases. To this end, we construct CodeTaste, a benchmark mined from large multi-file open-source refactorings. To score solutions, we combine repository test suites that measure functional correctness with tailored static checks that verify removal of undesired and introduction of desired code patterns using dataflow reasoning. Our results show a clear gap: agents perform well at implementing refactorings that are specified in detail, but often fail to discover the human refactoring choices when given a focus area for changes. A propose-then-implement decomposition improves alignment, and selecting the best-aligned proposal before implementation can yield further gains. CodeTaste provides an evaluation target and a potential preference signal for aligning coding agents with human refactoring decisions in realistic codebases. We release the benchmark, leaderboard, and code.

AI coding assistants can often write code that works, but working code is not always clean, easy to understand, or easy to extend and evolve. Human developers regularly improve code through refactoring: changing the structure of a program without changing what it does. We study whether today’s AI coding agents can perform these improvements at the scale of real software projects, and whether they can choose the same kinds of refactorings that human developers chose. To test this, we built CodeTaste, a benchmark of 100 large refactorings mined from open-source projects across six programming languages. Each task checks both whether the code still passes its tests and whether the agent made the intended structural changes. We find that agents can do reasonably well when given detailed refactoring instructions, but perform much worse when only told which part of the code needs improvement. Planning before editing helps, but current agents still fall far short of human refactoring judgment. This suggests that future coding agents need better taste, not just better code-writing ability.