When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs

Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evaluation of retrosynthesis performance remains limited. Existing benchmarks and metrics typically rely on published synthetic procedures and Top-K accuracy based on single ground-truth, which does not capture the open-ended nature of real-world synthesis planning. We propose a new benchmarking framework for single-step retrosynthesis that evaluates both general-purpose and chemistry-specialized LLMs using ChemCensor, a novel metric for chemical plausibility. By emphasizing plausibility over exact match, this approach better aligns with human synthesis planning practices. We also introduce CREED, a novel dataset comprising millions of ChemCensor-validated reaction records for LLM training, and use it to train a model that improves over the LLM baselines under this benchmark.

AI models are increasingly used to help design new molecules, but evaluating whether their proposed synthesis plans are useful remains difficult. Standard benchmarks usually check whether a model reproduces one recorded answer from a reaction database. This can be misleading, because in chemistry there are often many valid ways to make the same molecule. We introduce ChemCensor, a method for evaluating single-step retrosynthesis predictions by focusing on chemical plausibility rather than exact database matching. ChemCensor checks whether the key part of a proposed reaction and the surrounding chemical groups are supported by known reaction precedents. We also release new benchmark sets and a large reaction dataset, CREED, which is used to train a family of C3LM models. This work provides a more realistic approach to comparing synthesis planning systems. By rewarding chemically plausible alternatives rather than a single “correct” answer, the benchmark better reflects how chemists reason about possible routes and can help make AI tools for drug discovery more reliable.