Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings

Transformer-based architectures have advanced sequence modeling in language and vision, yet general-purpose representation learning for heterogeneous multivariate time series remains underexplored. We introduce CHARM (Channel-Aware Representation Model), which incorporates channel-level textual descriptions into a Transformer encoder equivariant to channel order. CHARM is trained with a Joint Embedding Predictive Architecture (JEPA) and a novel loss promoting informative, temporally stable embeddings; latent-space prediction encourages robustness to sensor noise while description-aware gating provides interpretability through learned inter-channel relationships. Across anomaly detection, classification, and short- and long-term forecasting, the learned embeddings achieve strong performance using only a linear probe. Performance is driven primarily by the JEPA objective and conditioning architecture, with text descriptions serving as channel identifiers for cross-dataset generalization.

Modern sensors generate vast amounts of time series data across industries like manufacturing, healthcare, and energy, but making sense of this data typically requires building separate, specialized models for each task. We introduce CHARM, a model that learns general purpose representations from multivariate sensor data by incorporating textual descriptions of what each sensor measures, similar to how a human expert interprets sensor readings by knowing the context of each measurement. CHARM is trained using a self supervised approach that predicts future patterns in a learned abstract space rather than raw noisy signals, making it robust to sensor noise. Our experiments show that CHARM's learned representations perform well across diverse downstream tasks, including anomaly detection, classification, and forecasting, using only lightweight task specific adapters, without retraining the full model for each new application.