A Bayesian Approach to Quantify the Uncertainty of Human Ratings in a Single-Instance Multimodal Framework

Human ratings are central to learning and inference across several application domains, but they are also subject to inter-rater biases and judgment errors. Quantifying the uncertainty of these human ratings would require repeated measurements, which are expensive and rarely available at scale. We propose a Bayesian graphical model to estimate the instance-level and item-level uncertainty of (subjective) human ratings by leveraging auxiliary (objective) data. Our model learns a shared latent content representation that explains factors common to both the human rating and auxiliary data and a latent uncertainty variable that captures fluctuations in the human assessments via a data-conditioned prior. We develop a scalable amortized variational inference procedure that uses modality-appropriate neural encoders and decoders to represent the posterior factors. Experiments on synthetic data demonstrate that our framework can accurately recover the latent uncertainty under targeted ablations and stress tests. We further demonstrate our approach on a real-world dataset of paired functional MRI scans and behavioral testing for autism, thus highlighting the need for uncertainty quantification.

Human ratings play a central role in decision-making pipelines across many application domains. For example, a clinician might rate the severity of an illness, which directly shapes downstream diagnosis and treatment plans. Despite their importance, human ratings are inherently subjective and can vary across people, contexts, and time. Thus, treating human ratings as reliable can lead to questionable downstream decisions. Conversely, if we can quantify the rating uncertainty, this information can be used to re-calibrate downstream tasks. We propose a novel Bayesian framework that estimates the item-level uncertainty of human ratings by leveraging auxiliary (objective) data collected in parallel, for example medical images for a patient. Our framework captures the common factors of both the subjective ratings and the auxiliary objective evidence to simultaneously estimate the uncertainty and identify the relevant auxiliary features that guide the estimation. We validate our framework on paired neuroimaging and behavioral data for autism. We demonstrate that our framework outperforms competing baseline methods and identifies meaningful brain influences associated with the disorder.