HEARTS: Benchmarking LLM Reasoning on Health Time Series

The rise of large language models (LLMs) has shifted time series analysis from narrow analytics to general-purpose reasoning. Yet, existing benchmarks cover only a small set of health time series modalities and tasks, failing to reflect the diverse domains and extensive temporal dependencies inherent in real-world physiological modeling. To bridge these gaps, we introduce HEARTS (Health Reasoning over Time Series), a unified benchmark for evaluating hierarchical reasoning capabilities of LLMs over general health time series. HEARTS integrates 16 real-world datasets across 12 health domains and 20 signal modalities, and defines a comprehensive taxonomy of 110 tasks grouped into four core capabilities: Perception, Inference, Generation, and Deduction. Evaluating 14 state-of-the-art LLMs on more than 20K test samples reveals intriguing findings. First, LLMs substantially underperform specialized models, and their performance is only weakly related to general reasoning scores. Moreover, LLMs often rely on simple heuristics and struggle with multi-step temporal reasoning. Finally, performance declines with increasing temporal complexity, with similar failure modes within model families, indicating that scaling alone is insufficient. By making these gaps measurable, HEARTS provides a standardized testbed and living benchmark for developing next-generation LLM agents capable of reasoning over diverse health signals.

Our bodies constantly produce signals over time — heartbeats, breathing, brain activity, sleep. AI systems like those behind chatbots are now used to interpret these signals, but whether they truly understand them or merely guess is unknown, and existing tests cover only a narrow range. So we built HEARTS, a large "exam" for AI spanning 16 real health datasets, 12 domains, 20 signal types, and 110 tasks, and used it to test 16 leading AI models. The results: AI performs far worse than specialized tools, often leans on simple shortcuts rather than step-by-step reasoning, and falters as signals grow more complex — and simply making models bigger does not help. HEARTS thus offers a shared yardstick for building AI that can truly reason about our health.