AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters

Text-to-image (T2I) systems increasingly rely on upstream prompters, either humans or multimodal large language models (MLLMs), to translate user intent into detailed prompts. Yet current benchmarks fix the prompt and only evaluate T2I models, leaving the prompting proficiency of this upstream component entirely unmeasured. We introduce AtelierEval, the first unified benchmark that quantifies prompting proficiency across 360 expert-crafted tasks. Grounded in a cognitive view, it spans three task categories and instantiates tasks using a taxonomy of real-world challenges, with a dual interface for both humans and MLLMs. To enable scalable and reliable evaluation, we propose AtelierJudge, a skill-based, memory-augmented agentic evaluator. It produces subjective and objective scores for prompt–image pairs, achieving a Spearman correlation of 0.79 with human experts, approaching human performance. Extensive experiments benchmark 8 MLLMs against 48 human users across 4 T2I backends, validate AtelierEval as a robust diagnostic tool, and reveal the superiority of mimicry over planning, advocating for an image-augmented direction for future prompters. Our work is released to support future research.

Text-to-image tools can now create impressive pictures from written instructions, but their results often depend heavily on who writes the prompt. Today, most benchmarks test the image generator itself, while ignoring the person or AI assistant that turns a user’s idea into a detailed prompt. We built AtelierEval, a benchmark for measuring this prompting ability in both humans and multimodal AI models. AtelierEval contains 360 carefully designed tasks, including open creative requests, tasks with strict visual constraints, and tasks where the goal is to imitate a reference image. To score the resulting prompts and images at scale, we also built AtelierJudge, an automated evaluator that checks both subjective qualities, such as visual appeal, and objective requirements, such as whether required objects or constraints are present. AtelierJudge closely matches expert human judgments, making large-scale comparison more practical. Using AtelierEval, we compared 8 multimodal AI models with 48 human users across 4 text-to-image systems. Our results show when AI assistants can outperform novice users, where skilled humans still matter, and why using reference images may be a promising direction for future prompting tools.