Sibyl: Temporal Backtesting for Literature-Based Scientific Discovery with Large Language Model Agents

We present Sibyl, a multi-agent LLM pipeline that autonomously mines scientific literature to generate falsifiable predictions, evaluated through a temporal backtesting framework analogous to quantitative finance. The system extracts structured claims from a training corpus (pre-cutoff publications), compiles a machine-readable knowledge base, generates testable hypotheses from identified gaps, and validates them against a held-out post-cutoff corpus. Applied to X-ray binary astrophysics as a proof-of-concept domain, the pipeline assembled 14,400 refereed papers, extracted over 11,000 structured claims, and generated 60 falsifiable predictions from pre-2015 literature alone. Of these, 11 (18%) were confirmed by independent post-2015 publications the system never observed. A post-hoc provenance audit identified three systematic failure modes - corpus contamination, validation-era leakage into hypothesis framing, and citation hallucination - the last of which we detected via a novel cross-prediction consistency check. Sensitivity analyses show that the confirmation rate is robust (12.5-18%) under progressively conservative filters. We present here the preliminary results from an ongoing project, introducing the pipeline architecture, the backtesting evaluation methodology, and the provenance audit protocol as contributions to the AI-for-science community.