Seizure-Semiology-Suite($S^3$): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in general video understanding, their capacity to interpret involuntary, and spatio-temporally evolving pathologic motor behaviors such as seizure semiology remains largely untested. To address this gap, we introduce Seizure-Semiology-Suite (S³), a clinically grounded dataset and benchmark for fine-grained, structured seizure semiology understanding. The dataset includes 438 seizure videos annotated with over 35,000 dense labels covering 20 ILAE-defined semiological features. Building on this dataset, we propose a seven-task hierarchical benchmark that systematically evaluates MLLMs from low-level visual perception to temporal sequencing, narrative report generation, and seizure diagnosis. To enable clinically meaningful evaluation of generated reports, we further introduce the Report Quality Index for Seizure Semiology (Seizure-RQI). Extensive baselines across 11 open-weight MLLMs reveal systematic weaknesses in laterality reasoning, temporal localization, symptom sequencing, and clinically faithful reporting. We show that seizure-specific fine-tuning substantially improves performance across tasks, and that a two-stage neuro-symbolic framework achieves an F1 score of 0.96 on epileptic versus non-epileptic seizure classification. Seizure-Semiology-Suite establishes a rigorous benchmark for evaluating multimodal models in safety-critical medical video understanding and guides the development of clinically reliable, domain-adaptive multimodal intelligence. Our code is publicly available at \href{https://github.com/LinaZhangUCLA/SeizureSemiologySuite}{SeizureSemiologySuite}.

Seizure videos contain important clues for diagnosis, but interpreting them requires clinicians to carefully observe how a person's movements, responses, and symptoms change over time. Recent AI systems that can "watch and describe" videos have raised hopes that they might one day assist this kind of clinical observation, but no one has carefully tested whether they actually understand what is happening in a seizure video. We address this question by building Seizure-Semiology-Suite, a clinically annotated collection of 438 seizure videos with detailed labels for 20 seizure-related signs. We also design an evaluation framework that tests whether AI models can recognize seizure signs, track when and where symptoms appear, describe how a seizure evolves, write clinical-style reports, and help distinguish epileptic from non-epileptic seizures. Using this benchmark, we find that current AI models can detect some obvious movement patterns, but they often struggle with details that matter in clinical practice, such as which side of the body is affected, timing, symptom order, and faithful reporting. We further show that training models on seizure-specific data and using a two-stage classification method can substantially improve seizure classification performance. Ultimately, our work provides a rigorous standard for measuring progress, helping ensure that future AI tools become reliable enough to assist doctors in analyzing medical videos.