Identifying Metric Structures of Deep Latent Variable Models

Deep latent variable models learn condensed representations of data that, hopefully, reflect the inner workings of the studied phenomena. Unfortunately, these latent representations are not statistically identifiable, meaning they cannot be uniquely determined. Domain experts, therefore, need to tread carefully when interpreting these. Current solutions limit the lack of identifiability through additional constraints on the latent variable model, e.g. by requiring labeled training data, or by restricting the expressivity of the model. We change the goal: instead of identifying the latent variables, we identify relationships between them such as meaningful distances, angles, and volumes. We prove this is feasible under very mild model conditions and without additional labeled data. We empirically demonstrate that our theory results in more reliable latent distances, offering a principled path forward in extracting trustworthy conclusions from deep latent variable models.

Reducing complex phenomena to a smaller number of meaningful factors has always been central to scientific discovery. Today, this process is increasingly driven by machine learning models rather than individual intuition. These models are highly effective at predicting the studied phenomena while relying on some notion of a hidden structure. However, the structures the models produce are fundamentally ambiguous which makes their interpretation difficult.A key issue is that many different internal representations, ways the model “understands” the data, can perform equally well. This means that even when models appear to succeed, the specific factors they rely on might be arbitrary or misleading. As a result, two research teams using similar tools on the same dataset can reach very different conclusions based on the recovered representations. This lack of identifiability undermines trust in hidden mechanisms recovered by machine learning algorithms and limits their usefulness for scientific insight.Our work takes a different perspective. Rather than trying to make the internal representations themselves uniquely defined, we focus on relations like distances, angles, and volumes as this kinds of information os often what matters most in scientific exploration. We prove that such geometric relations between the learned representations can, in fact, be identified reliably, when their measurement respects the underlying geometry of the model.