The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation

Public leaderboards rank coding agents primarily by which model they use, treating the surrounding agent harness (the scaffold that issues tools, manages context, and decides when to stop) as a controlled constant. We show this assumption breaks down empirically. Running two recent agentic coding models (Qwen 3.6 Plus and MiniMax M2.5) across three open-source harnesses (Goose, OpenCode, OpenHands-SDK) on a stratified 50-task subset of Terminal-Bench Pro, we find that harness choice produces up to a $\mathbf{40\times}$ difference in tokens consumed per solved task while paired pass-rate differences within a model remain within 0-8 percentage points (95% paired-task bootstrap CIs include zero for all but the largest pairwise gap). The same harness-specific failure fingerprints (REASON-dominated for Goose, VERIFY-and-MAX_TURNS for OpenHands-SDK, idle-loop / TIME for OpenCode) replicate independently on both models, identifying these patterns as scaffold properties rather than model properties. We argue that, for human-centered coding-agent evaluation, "model name" is an incomplete unit of comparison: harness-model pairs determine real-world cost, latency, and the oversight burden a developer must absorb (no-action turns are a per-task wait tax, not just a token tax). We include anonymized configurations, raw trial logs, aggregated snapshots, and analysis scripts as supplementary material.