Predictable Compression Failures: Order Sensitivity and Information Budgeting for Evidence-Grounded Binary Adjudication

Transformers used for evidence-grounded binary adjudication (e.g., support/refute, yes/no, or verifier-backed pass/fail decisions) can be sensitive to the order in which exchangeable evidence is presented, producing dispersion across permutations and unreliable attempted answers under a verifier-relative Bernoulli predicate. We treat evidence order as a nuisance variable and formalize an expectation–realization gap: next-token training can minimize expected conditional description length over orderings while a fixed ordering remains position-sensitive. Our Quantified Martingale Violation (QMV) bound predicts the dispersion induced by adjacent-rank positional sensitivity, with $O(\log n)$ growth in the harmonic regime; our Expectation-level Decompression Law (EDFL) specializes a KL convexity/data-processing bound to Bernoulli predicates, yielding Bits-to-Trust (B2T), Risk-of-Hallucination (RoH), and an Information Sufficiency Ratio (ISR) gate for answer/abstain decisions. On 3,059 grounded items from FEVER, HotpotQA, NQ-Open, PopQA, and Controls, we observe logarithmic dispersion and positive Jensen gains from uniform permutation mixtures. In one pre-specified held-out audit (528 items), the analytically fixed $ISR=1$ gate attains 0.0–0.7% hallucination with 20.6–27.9% abstention (95% CIs), supporting the operating point without claiming universal calibration across all model families or unrestricted generation.

AI systems are often asked to answer factual questions using a set of retrieved evidence. We found that, even when the evidence is exactly the same, changing the order in which it is shown can change whether a model seems confident, uncertain, or wrong. This is a problem because the truth of a claim should not depend on whether the relevant evidence appears first, last, or in the middle. Our paper studies this failure mode in settings where the answer can be checked, such as support/refute decisions, yes/no questions, multiple-choice answers, or pass/fail tests. We measure how much a model’s answer changes across harmless reorderings of the same evidence, then use this to decide whether the model has enough support to answer. If the evidence looks sufficient, the system answers; if not, it abstains or asks for more evidence. Across several factual question-answering benchmarks, this approach exposed predictable order sensitivity and greatly reduced unsupported answers, at the cost of abstaining on some cases. The goal is not to solve all hallucination in open-ended generation, but to make checkable AI decisions more reliable by teaching systems when not to guess.