DevBench: An Interaction-Grounded Benchmark for Code Completion Models

Code completion is the most frequent form of human-AI interaction in modern software development: a developer positions their cursor, and the model suggests what comes next. Yet existing benchmarks for this critical interaction are not grounded in how developers actually use completion tools. DevBench is an interaction-aware benchmark derived from over one billion real developer code completion interactions, capturing the scenarios, cursor positions, and difficulty patterns where current models succeed and fail in practice. It includes 1,800 evaluation instances across six programming languages and six empirically derived task categories, synthesized using multiple generator model families to ensure contamination resistance. The evaluation combines functional correctness (Pass@1), similarity-based metrics, and LLM-judge assessments of contextual relevance and helpfulness, mirroring the distinction developers themselves make between "correct" and "useful" completions. Evaluating 9 state-of-the-art models, the strongest achieves only 43.5\% Pass@1, and models that score highest on perceived helpfulness often fail hidden functional assertions, revealing a tension directly relevant to developer trust. Our benchmark provides fine-grained, cross-language diagnostics to guide improvement of the completion models that power everyday developer-AI collaboration.