OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data

Large Language Models have shown strong capabilities in interpreting multimodal data but remain limited in handling time-series data. Addressing this gap could help to translate longitudinal and wearable data into actionable insights and patient-facing digital health applications. We propose OpenTSLM, an open-source family of Time Series Language Models integrating time-series as a native modality into pretrained LLMs, enabling natural-language prompting and reasoning over multiple time-series via either soft prompting (OpenTSLM-SoftPrompt) or cross-attention (OpenTSLM-Flamingo). To enable training models for time-series reasoning, we introduce three datasets: HAR-CoT (human activity recognition), Sleep-CoT (sleep staging), and ECG-QA-CoT (electrocardiogram question answering). Across tasks, OpenTSLM models outperform baselines, reaching F1 scores of 69.88% in sleep staging and 67.64% in HAR; OpenTSLM-Flamingo also scales more efficiently in memory as the number and length of time series increase. Expert evaluations with cardiologists show that OpenTSLMs exhibit strong reasoning capabilities on raw ECG data.

In this work, we address a major gap in today’s AI systems: their limited ability to understand sensor data that changes over time. Since chat-based AI tools (Large Language Models) like ChatGPT became widely available in 2022, people have used them to write, summarize, program, and answer questions in natural language. Soon after, these systems also became capable of processing images, video, and audio, for example, by describing visual content or transcribing speech into text. However, they still struggle with another common kind of data: time-series signals, such as heart activity, sleep recordings, movement sensors, or measurements collected continuously from machines and the human body. We introduce OpenTSLM, an open-source family of models that can use time-series data as a native input. Our models combine raw signals with additional context through text and provide answers and explanations in natural language. We test this approach on human activity recognition, sleep staging, and electrocardiogram questions and achieve strong results. We also evaluate the model’s electrocardiogram explanations with expert clinicians from Stanford Hospital, whose assessment was largely positive regarding their clinical relevance and usefulness. Since many important decisions in medicine and beyond depend on understanding how signals change over time, we expect that OpenTSLM is a step toward AI systems that can help people interpret complex sensor data through ordinary language and support tasks such as monitoring, diagnosis, and decision-making under human oversight.