A Human in the Loop Is Not a Human in Charge: When Discretion Becomes Ratification in Algorithm-Assisted Public Decisions
Abstract
Human-in-the-loop review is widely treated as the safeguard for algorithmic decision-support in the public sector, but formal involvement does not by itself preserve independent judgment. We introduce the optimisation trap, a workflow-level account of how the timing of algorithmic advice, information compression, and workload can thin substantive review while leaving formal authority intact. We test the mechanism in a welfare-eligibility simulation that separates case judgment from workflow dynamics: four open-source LLM agents make review decisions, while a mathematical layer tracks attention allocation and trust updating. Across 12 runs, 120 agent-days, and 16{,}070 decisions, showing the algorithmic recommendation before a provisional assessment is formed reduces routine-case error correction by 16.7 to 43.1 percentage points across all four models. Higher routine algorithmic accuracy deepens edge-case dependence only when algorithmic advice appears first, and the resulting attention shifts do not reliably improve accuracy for documentation-gap and gig-income applicants. The simulation results should be read as directional evidence from LLM-based simulation, not as estimates of human behaviour. The contribution is to generate testable hypotheses for human experiments and workflow audits.