Paper #22: AutoFiction: Measuring AI ability to execute long-horizon writing tasks

AI systems are increasingly capable of long-horizon narrative planning and generation, but existing creative writing evaluations mostly focus on short-form outputs, which provides little evidence about whether models can execute book-length writing tasks. We introduce AutoFiction, an open-source research platform and agent harness for generating and evaluating long-form AI fiction. AutoFiction currently hosts 41 AI-generated books across diverse genres and collects human feedback through book reviews, chapter comments, paragraph-level annotations, and engagement signals (e.g. reading time). Initial human feedback reveals that AI-written novels contain interesting descriptions and concepts, but often fail in long-range coherence and prose quality, including conceptual repetition, distracting details, and weak dialogues. Rather than endorsing AI-written fiction, AutoFiction serves as a transparent platform for tracking what these systems can and cannot yet do in terms of long-form narrative generation.