Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces

The Platonic Representation Hypothesis posits that learned representations from models trained on different modalities converge to a shared latent structure of the world. However, this hypothesis has largely been examined in vision and language, and it remains unclear whether time series participate in such convergence. We first examine this in a trimodal setting and find that independently pretrained time series, vision, and language encoders exhibit near-orthogonal geometry in the absence of explicit coupling. We then apply post-hoc alignment by training projection heads over frozen encoders using contrastive learning, and analyze the resulting representations with respect to geometry, scaling behavior, and dependence on information density and input modality characteristics. Our investigation reveals that overall alignment in contrastive representation spaces improves with model size, but this alignment is asymmetric: time series align more strongly with visual representations than with text, and images can act as effective intermediaries between time series and language. We further see that richer textual descriptions improve alignment only up to a threshold; training on denser captions does not lead to further improvement. Analogous effects are observed for visual representations. Our findings shed light on considerations for building multimodal systems involving non-conventional data modalities beyond vision and language. Code and data are available at https://github.com/Rose-STL-Lab/tvl-alignment.

AI systems are increasingly being developed to understand information in different forms, such as numbers, images, and language. Recent studies suggest that well-trained models can develop similar internal ways of representing the world, even when they are trained on different types of data. We ask whether this is also true for time series data, which represents information as values changing over time, such as stock prices or heart rate measurements. We study whether models trained on time series, images, and text naturally learn compatible internal representations. We find that they do not; without explicit training to connect them, these models represent the same information in very different ways. When we train them to align using a learning approach that encourages related inputs to be mapped together, we find that time series align more easily with visual plots of the same data than with written descriptions. This is interesting because the models are not told that plots are direct visual versions of the same underlying data; they must discover this relationship from data alone. We also find that adding more detailed text helps only up to a point, after which it no longer improves alignment. Finally, we observe that images can help connect time series and language, acting as a bridge between the two. Our results provide insights for building AI systems that can work across numerical data, images, and language more effectively.