Weak Diffusion Priors Can Still Achieve Strong Inverse-Problem Performance

Can a diffusion model trained on bedrooms recover human faces? Diffusion models are widely used as priors for inverse problems, but standard approaches usually assume a high-fidelity model trained on data that closely match the unknown signal. In practice, one often must use a mismatched or low-fidelity diffusion prior. Surprisingly, these weak priors often perform nearly as well as full-strength, in-domain baselines. We study when and why inverse solvers are robust to weak diffusion priors. Through extensive experiments, we find that weak priors succeed when measurements are highly informative (e.g., many observed pixels), and we identify regimes where they fail. To explain this behavior, we combine Bayesian-consistency theory with local-correlation analysis: the theory gives conditions under which high-dimensional measurements make the posterior concentrate near the true signal, while the correlation analysis shows that weak and stronger natural-image priors can share similar local spatial structure. These results provide a principled justification on when weak diffusion priors can be used reliably. Code is available at https://github.com/jjia131/weak-diffusion-priors-inverse-problem.

This paper studies whether a weak diffusion model can still help recover a clean image from a noisy or incomplete one. Surprisingly, the answer is often yes. Even when the diffusion model is low quality or trained on a different type of image, it can still give strong reconstructions when the observed data contain enough information. The main reason is that the data can guide most of the recovery, while the diffusion model supplies useful local image patterns, such as nearby pixel structure. The paper also shows when this idea fails: if too much information is missing, such as in large missing regions or very hard super-resolution tasks, a stronger and better-matched model becomes much more important.