CAD-bench: Functional Failure Modes in CAD-Generating Agents

Language-model agents are increasingly able to operate computer-aided design (CAD) toolchains, but their failures are often obscured by surface-level success signals. A generated CAD artifact can compile, render, and resemble the requested object while still failing the mechanical requirements of the task. We introduce CAD-bench, an execution-based benchmark for exposing functional failure modes in CAD-generating agents. CAD-bench contains 17 tasks across four difficulty tiers, ranging from basic solids to threaded mating pairs and functional gear trains. Each submission is executed, exported as geometry, and evaluated with task-specific checks, including dimensional and pose verification, reference-geometry gates, thread-profile analysis, and Blender-based rigid-body simulation. The benchmark supports both one-shot CAD-code generation and agent harnesses that produce final STEP artifacts in an executable environment. Initial results show that CAD-bench is not saturated by current systems. The strongest standalone model reaches 59.9\% overall, while functional tasks remain near zero for most standalone runs. Agent harnesses perform better than one-shot generation, but still fail frequently on interfaces, standards-like details, and mechanisms. Reported agent submissions build the right-angle gearbox in 96.9\% of attempts but average only 4.0\% on the full functional score. CAD-bench therefore isolates a recurring failure mode for artifact-producing agents: executable output is not equivalent to mechanically meaningful task success.