BEDTime: A Unified Benchmark for Automatically Describing Time Series

Recent works propose complex multi-modal models that handle both time series and language, ultimately claiming high performance on complex tasks like time series reasoning and cross-modal question answering. However, they skip foundational evaluations that such complex models should have mastered. So we ask a simple question:$\textit{How well can recent models describe structural properties of time series?}$ To answer this, we propose that successful models should be able to $\textit{recognize}$, $\textit{differentiate}$, and $\textit{generate}$ descriptions of univariate time series. We then create $\textbf{BEDTime}$, a benchmark to assess these novel tasks, that comprises $\textbf{five datasets}$ reformatted across $\textbf{three modalities}$. In evaluating $\textbf{17 state-of-the-art models}$, we find that (1) surprisingly, dedicated time series-language models fall short, despite being designed for similar tasks, (2) vision language models are quite capable, (3) language only methods perform worst, despite many lauding their potential, and (4) all approaches are clearly fragile to a range of real world robustness tests, indicating directions for future work. Together, our findings critique prior works' claims and provide avenues for advancing multi-modal time series modeling.

A time series is a sequence of numbers collected over time, like a patient's heart rate or daily stock prices. They are everywhere in medicine, finance, and engineering. Recent Language Models (LMs) claim to analyze and reason about these patterns using natural language, but they're typically tested on complicated tasks before anyone checks whether they can handle the basics. So, we asked a simple question: can these models actually describe what a time series looks like? To find out, we built BEDTime, a unified test that asks 17 leading LMs to do three things: recognize whether a description matches a time series, differentiate the best description from multiple choices, and write a description of a given time series. We assembled over 40,000 examples from five existing datasets and tested the models across three different modalities. The findings were revealing. Models that see time series as images performed best, while text-only approaches struggled, and specialized "time series language models" underperformed expectations. Every model broke down when we added realistic challenges like noise or missing values, exposing how fragile current systems really are. This is the gap BEDTime addresses: it contributes the first unified benchmark for this foundational skill, letting researchers directly compare models, diagnose where they fail, and track real progress on describing time series.