Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics

Image-to-image relighting requires representations that separate illumination from scene properties while preserving dense geometry, material, and photometric cues. We use this task as a probe of visual priors: unlike recognition tasks that reward invariance, relighting tests whether visual features retain the information needed for light transfer. Through a controlled generative relighting framework, we find that strong semantic encoders can degrade relighting quality, exposing a semantic--photometric trade-off between abstraction and physical fidelity. We introduce \textbf{Augmented Latent Intrinsics (ALI)}, which balances this trade-off by fusing dense, pixel-aligned visual features into a latent-intrinsic relighting model and refining it with self-supervision on unlabeled real image pairs. ALI improves relighting quality, especially on glossy, metallic, and transparent materials, and demonstrates that generative relighting is an effective tool for quantifying what visual encoders encode about the physical world.

Relighting an image — taking a photo and changing how it's lit, as if the sun moved or a lamp was switched on — sounds like a simple visual edit, but it's one of the hardest problems in computer vision. To do it convincingly, a model has to figure out what's a shadow versus what's a dark surface, what's a highlight versus what's a white object, and how light would bounce off skin, metal, glass, or fabric. In other words, relighting demands a genuine understanding of a scene's geometry, materials, and semantics, not just what objects are present. In this paper, we argue that this difficulty is exactly what makes relighting useful as a scientific tool. Beyond the obvious goal of producing better-looking relit images, we use relighting as a lens — a way to probe how much physical understanding is actually baked into modern AI vision models. If a model's internal representation truly captures the 3D structure of a scene and how surfaces respond to light, relighting should work well. If the representation only captures high-level concepts ("this is a face," "this is a chair"), relighting will fall apart on the details. What we found is surprising. Popular vision encoders like CLIP and DINO, which excel at recognizing and categorizing objects, often hurt relighting performance. Their training pushes them to ignore exactly the kind of fine-grained lighting and texture cues that relighting depends on. Meanwhile, encoders trained to reconstruct pixels or to preserve dense spatial detail do far better, even though they score lower on standard recognition benchmarks. This reveals a trade-off that the field has largely overlooked: the features that make a model good at "knowing what things are" are not the same features that make it good at "knowing how things look under light." Building on this insight, we introduce Augmented Latent Intrinsics (ALI), a relighting system designed to balance semantic understanding with fine photometric detail. ALI achieves state-of-the-art results among open-source relighting methods, with particularly strong gains on tricky materials like glossy and reflective surfaces. More broadly, our work suggests that relighting is not just a creative tool for photographers and filmmakers — it's a sensitive diagnostic for whether AI vision systems actually understand the physical world they're looking at.