Personalized Image Generation via Human-in-the-loop Bayesian Optimization

Imagine Alice has a specific image $x^\ast$ in her mind, say, the view of the street in which she grew up during her childhood. To generate that exact image, she guides a generative model with multiple rounds of prompting and arrives at an image $x^{p*}$. Although $x^{p*}$ is reasonably close to $x^\ast$, Alice finds it difficult to close that gap using language prompts. This paper aims to narrow this gap by observing that even after language has reached its limits, humans can still tell when a new image $x^+$ is closer to $x^\ast$ than $x^{p*}$. Leveraging this observation, we develop **MultiBO** (Multi-Choice Preferential Bayesian Optimization) that carefully generates $K$ new images as a function of $x^{p*}$, gets preferential feedback from the user, uses the feedback to guide the diffusion model, and ultimately generates a new set of $K$ images. We show that within $B$ rounds of user feedback, it is possible to arrive much closer to $x^\ast$, even though the generative model has no information about $x^\ast$. Qualitative scores from $30$ users, combined with quantitative metrics compared across $5$ baselines, show promising results, suggesting that multi-choice feedback from humans can be effectively harnessed for personalized image generation.

Imagine trying to recreate a very specific memory, like the exact street you grew up on as a child. Even with today’s AI image generators, it can be hard to describe every detail perfectly using words alone. You might get close, but not quite capture what you truly have in mind. This work introducesa a new system called MultiBO that helps people guide AI image generation beyond what language alone can express. Instead of relying only on text prompts, the system shows the user several new image options at each step and asks which ones feel closer to the image they are imagining. The AI then learns from these choices and generates better options in the next round. Over multiple rounds of feedback, the system gradually moves closer to the user’s ideal image, even though the AI never actually sees the original image in the person’s mind. Tests with 30 users showed that this interactive approach helped create more personalized and satisfying images compared to existing methods.