Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation

Manufacturable chip layouts must satisfy thousands of geometry-based design rules, and design rule checking (DRC) enforces them by running executable DRC scripts on layouts. Translating natural language rules into correct DRC scripts is labor-intensive and requires specialized expertise, motivating LLM agents for DRC script synthesis and debugging. However, existing benchmarks have small evaluation sets and often evaluate scripts by code similarity rather than execution correctness, and prior machine learning-based methods either ignore execution feedback or require labeled test layouts as agent's input. To this end, we introduce Rule2DRC, a large-scale benchmark for DRC script coding agents with 1,000 rule-to-script tasks and 13,921 evaluation chip layouts for execution-based scoring. Rule2DRC provides an evaluation pipeline that measures functional correctness via DRC execution outcomes without requiring evaluation layouts as input to the agent. We also propose SplitTester, a tester agent for program selection that uses execution feedback to generate discriminative test cases and separate previously indistinguishable candidate scripts, substantially improving Best-of-N selection performance in this domain. We release the code at https://github.com/snu-mllab/Rule2DRC.

Computer chips are built from layouts that must follow thousands of geometric design rules to actually be manufacturable. Engineers verify these rules using special checking scripts, but writing those scripts by hand from written rule descriptions takes a lot of time and specialized knowledge. Large language models could potentially automate this work, but the current ways of measuring how well they do have two main weaknesses. The evaluation sets tend to be small, and they often judge a script by how similar its code looks to a reference answer rather than by whether the script actually runs and produces the right results. To address this, we built Rule2DRC, a large benchmark containing 1,000 tasks that ask an AI to translate a written rule into a working checking script. It also includes 13,921 chip layouts used to grade those scripts based on what happens when they are run on real designs. The grading pipeline measures true functional correctness without ever handing the test layouts to the AI as part of its input. We also created SplitTester, a helper agent that uses execution feedback to design new test cases aimed at exposing differences between candidate scripts that otherwise appear equally good. This makes it much easier to pick the best script out of a batch of generated options.