Escaping Whack-a-Mole: Optimizing Documentation as Repo-Specific Playbooks for Coding Agents

As large language models increasingly function as autonomous coding agents, code documentation should be designed not for human readability, but for agent executability — serving as repo-specific playbooks that specify precise behaviors agents can follow. We formulate agent-oriented documentation generation as a black-box optimization problem over the documentation space, where quality is defined solely by downstream code correctness. A central challenge for conventional LLM refinement methods is output coupling—program entities are interdependent, and refining the documentation of one entity can invalidate its callers, resulting in a persistent whack-a-mole phenomenon during inference-time scaling. We propose DocSearch, a dependency-guided bi-level search framework that systematically exploits test-time feedback. The outer level conducts a priority search over the program-entity dependency DAG, enforcing a callee-before-caller refinement order to prevent downstream interference. The inner level performs a beam search over documentation refinements, using diversified error message sampling from self-generated unit tests to better exploit diagnostic signals and escape local optima. On DevEval+, DocSearch achieves 90.7\% solve rate with GPT-4o, outperforming the strongest baseline by 32.6\%. Cross-language experiments further demonstrate that optimized documentation transfers effectively to different target programming languages. Code is available at https://github.com/ccsnow127/docsearch.

Coding agents can in principle automate software tasks like legacy modernization, where the goal is to recreate an existing codebase in a different language. But source code is bulky and represents just one specific way of achieving a behavior, entangled with the quirks of the language and the implementation-level design. What the agent actually needs is a high-level description of the repository's behavior — one that transfers cleanly to any target language or framework. We call this a repository-specific playbook: precise, executable documentation written for the agent rather than for a human reader. We formulate the generation of such documentation as a black-box optimization problem and solve it with a dependency-guided search that uses feedback from auto-generated tests. The central difficulty is output coupling: refining one component's documentation often invalidates the agent's reproduction of components that depend on it, producing a "whack-a-mole" effect that stalls prior iterative refinement methods. Our method finalizes components in reverse topological order under a worthiness criterion that prevents regressions, yielding monotonic improvement.