Poster

OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data

Patrick Langer ⋅ Thomas Kaar ⋅ Max Rosenblattl ⋅ Maxwell Xu ⋅ Winnie Chow ⋅ Martin Maritsch ⋅ Robert Jakob ⋅ Ning Wang ⋅ Juncheng Liu ⋅ Aradhana Verma ⋅ Brian Han ⋅ Daniel Kim ⋅ Henry Chubb ⋅ Scott Ceresnak ⋅ Aydin Zahedivash ⋅ Alexander Sandhu ⋅ Fatima Rodriguez ⋅ Daniel McDuff ⋅ Elgar Fleisch ⋅ Oliver Aalami ⋅ Filipe Barata ⋅ Paul Schmiedmayer

Abstract

Large Language Models (LLMs) have shown strong capability in interpreting multimodal data but remain limited in their ability to natively handle time-series data. Addressing this limitation could enable the translation of longitudinal and wearable sensing data into actionable insights and patient-facing digital health applications. We propose OpenTSLM, a family of Time Series Language Models that integrate time-series as a native modality into pretrained LLMs, enabling natural-language prompting and reasoning over multiple time-series. We implement two OpenTSLM variants based on soft prompting (OpenTSLM-SP) and cross-attention (OpenTSLM-Flamingo). To conduct comprehensive experiments on reasoning over medical text and time-series, we introduce three chain of thought (CoT) datasets: HAR-CoT (human activity recognition), Sleep-CoT (sleep staging), and ECG-QA-CoT (electrocardiogram question answering). Across tasks, OpenTSLM models consistently outperform baselines. OpenTSLMs with time-series encoders trained from scratch achieve 69.88% in sleep staging and 65.44% in HAR, while OpenTSLM combined with time series foundation models (TSFMs) achieve 68.33% and 67.64%, compared to 9.05% and 60.44% for fine-tuned text-only baselines. Additionally, we conduct expert evaluations with cardiologists, which show that OpenTSLMs exhibit strong reasoning capabilities and temporal understanding on raw sensor data for ECG-QA. We further show that OpenTSLM-Flamingo models scale better in memory as the number and length of time-series increase. To facilitate further research, we release all code, datasets, and models as open-source resources.