Spectral Evolution Search: Efficient Inference-Time Scaling for Reward-Aligned Image Generation

Inference-time scaling offers a flexible way to align visual generative models with downstream objectives without updating model parameters. In modern image generation models, a natural way to do this is to optimize the random noise from which generation starts. However, searching in this high-dimensional noise space is highly inefficient, because many directions have little effect on the final image. We trace this inefficiency to a spectral bias in generative dynamics: model sensitivity to initial perturbations decays rapidly as frequency increases. Based on this insight, we propose Spectral Evolution Search (SES), a plug-and-play framework for initial noise optimization that performs gradient-free evolutionary search in a low-frequency subspace. Theoretically, we derive the Spectral Scaling Prediction from perturbation propagation dynamics, which explains the frequency-dependent impact of perturbations. Extensive experiments across diverse settings show that SES substantially improves the trade-off between generation quality and computational cost, consistently outperforming strong baselines under the same compute budget.

Modern image generation systems can create impressive pictures from text. However, it is often difficult to make them better satisfy a specific goal. For example, users may want images that match a prompt more closely, look more beautiful, or better reflect human preferences. A simple way to improve results at inference time is to generate many samples and pick the best one. However, this approach can be wasteful because it does not use the computation budget efficiently. Our key finding is that not all changes to the random starting signal matter equally. Low-frequency patterns have a much stronger effect on the final image. Small details with high frequency matter much less. Based on this insight, we design a search method that focuses only on the most important part of the starting signal. Instead of searching blindly over the full space, it searches where changes matter most. Across several image generation models and evaluation goals, our method consistently finds better images under the same computation budget. This makes test-time improvement of image generation more practical, flexible, and efficient.