PepLang-Bench - Evaluating Large Language Models Understanding On Peptide Related Tasks

Large language models are increasingly applied to chemistry, yet their capabilities on peptide-specific tasks, which require integrating sequence conventions, chemical notation systems, and structural reasoning, remain untested. We introduce PepLang-Bench, a benchmark of seven tasks across three categories: property prediction (GRAVY, isoelectric point, instability index), notation conversion (SMILES-to-HELM and reverse), and structural adaptation (linear peptide to cyclized SMILES and reverse), evaluated on both therapeutic and synthetic peptide datasets. We benchmark eight models spanning generalist (GPT-5, o4-mini, Gemini-3-Pro, Qwen 3.5) and domain-specific (ChemDFM-R, ChemDFM-v2.0, Ether-0, NatureLM-8x7B) large language models. GPT-5 with code interpreter achieves 64\% in isoelectric point prediction by autonomously invoking required domain-specific knowledge, while the same model without tools scores 21.1\%. Domain-specific models score 0\% in all notation and structural adaptation tasks despite reporting confidence above 0.92. Synthetic peptides containing D-amino acids and non-canonical residues further degrade performance of all models, exposing a distribution gap that current chemistry-focused pre-training does not address. PepLang-Bench establishes peptides as an open challenge for LLM reasoning and provides verifiable reward functions that can serve as training signals for future peptide-specific reasoning models via reinforcement learning.