Audit-Grade Harness for Agent-driven Scientific Computation Workflows

LLM agents are increasingly used to drive scientific computation workflows — code generation, simulation execution, reference-data acquisition, and analysis — across fluid dynamics, photonics, biomedicine, chemistry, and materials science. Reproducibility concerns familiar from human-led empirical science re-emerge in compressed form when a single agent is responsible for design, execution, and reporting; an independent evaluation of Sakana's AI Scientist found a 42% experiment-failure rate and "hallucinated numerical results" in generated manuscripts, illustrating how internal validation alone may not suffice. We contribute a four-component audit-grade harness around an LLM-agnostic scientific coding agent at supervised autonomy: a registry of version-pinned, sandboxed containerized scientific services for the agent's tool surface; a managed cloud-compute layer that lets simulations run at the paper's intended fidelity rather than at local-machine fidelity; a durable per-action provenance log; and a fresh-context verifier whose model identity is decoupled from the executor's by configuration. Each component addresses a recurring failure mode catalogued in earlier work — Research-Execution Disconnection (RED), where the agent researches documentation but fails to use the findings in code; Sophisticated Fabrication (SF), where the agent produces plausible synthetic data when an external source returns an error; and Silent Degradation (SD), where the agent substitutes a simplified implementation without surfacing the substitution — observed across diverse natural-science domains. We use published-paper reproduction as the testbed for verifiable discovery and deploy the harness on three case studies: a multi-zone metasurface in-coupler in photonics; a computational fluid dynamics (CFD) reproduction of a data-center thermal study at the published mesh resolution and solver settings; and a BRCA1 fitness–structure mapping with phased verification.