PromptSplit: Revealing Prompt-Level Disagreement in Generative Models

We study the problem of comparing prompt-conditioned generative models through the lens of kernel-based distribution comparison. Given two models evaluated on prompts, our goal is to identify prompt regions where their conditional output distributions differ, without supervision or repeated sampling for each prompt. We propose \emph{PromptSplit}, an unsupervised spectral kernel method that represents each prompt--response pair through a tensor-product feature map and compares models via the difference of their joint prompt--output kernel covariance matrices. The leading positive eigendirections of this covariance difference provide soft modes of discrepancy, allowing the method to localize prompt categories associated with systematic differences in model behavior. We show that the required eigenspectrum can be computed through an equivalent block kernel matrix involving Hadamard products of prompt and output kernels, and introduce a random-projection implementation that reduces the computation to $O(r^2\cdot\max\{n,r\})$ for projection dimension $r$. We further prove that the projected formulation provides a controlled approximation to the full spectral comparison. Our numerical evaluation on controlled and real prompt-guided generation tasks, including text-to-image and text-to-text models, shows that PromptSplit recovers known prompt-dependent differences and reveals explainable modes of difference between generative models.