Beyond Generative Priors: Minority Sampling with JEPA-Guided Diffusion

Minority sampling aims to generate low-density instances on a data manifold and is of central importance in applications such as medical diagnosis, anomaly detection, and creative AI. Existing approaches, however, define minority samples relative to generative priors learned from training data, confining rarity to model-specific notions that may poorly reflect real-world semantics. In this work, we propose a world-centric perspective on minority sampling, which defines rarity with respect to real-world priors rather than generator-induced densities. To this end, we introduce \emph{JEPA guidance}, a diffusion sampling framework guided by a Joint-Embedding Predictive Architecture (JEPA)---a class of world models that encode broad, semantically rich representations. JEPA guidance steers diffusion trajectories toward low-density regions under the implicit density induced by the JEPA, thereby aligning generated minorities with real-world semantic rarity. To make JEPA guidance computationally practical, we develop principled approximation strategies accompanied by theoretical error bounds, significantly reducing the overhead of guidance computation. Extensive experiments across unconditional, class-conditional, and text-to-image generation demonstrate that JEPA guidance consistently improves the fidelity and semantic validity of minority samples, outperforming generator-centric baselines in capturing real-world notions of rarity. Code is available at https://github.com/soobin-um/jepa-guidance.

When generating images with AI, some types of images are naturally rare, such as unusual animal behaviors or atypical scenes. Existing methods try to generate such rare images, but they can only find what is rare according to their own training data, which may not reflect what is truly rare in the real world. We propose a new approach that uses a separate AI model, trained on a much larger and more diverse set of images, to guide the image generation process toward producing samples that are genuinely rare in a broader, real-world sense. To make this practical, we develop efficient computational techniques with theoretical guarantees. Our experiments show that this approach consistently produces high-quality rare images across various generation settings, outperforming existing methods in capturing real-world rarity.