UniDesigner: Language Models as Unified Planners for Agentic Design

We cast agentic design as a planning problem in which a language model (LM) acts as a domain-agnostic planner: given a natural-language design goal, it must emit a structured, tool-executable plan—an Intermediate Representation (IR) such as an XML stroke program, a JSON garment specification, or a Blender Python script—that an external synthesis tool can validate and enact. Viewed this way, existing LM-based design agents resemble isolated, domain-bound planners trained by imitation (supervised fine-tuning), which leaves them fragile at plan feasibility, goal satisfaction, and the soft preferences that make designs useful. We propose UniDesigner, a unified framework that treats heterogeneous design domains—sketch drawing, clothing design, and 3D object generation—as instances of a common LM-based planning problem specified by an IR schema, a synthesis tool (executor/validator), and a design query (goal). To align the planner with the partial-order structure of design objectives, we introduce IR Preference Optimization (IRPO), a hierarchy-aware preference optimization method that places plan feasibility above goal achievement, and goal achievement above aesthetic soft preferences. Experiments across the three design domains show that UniDesigner optimized with IRPO consistently yields more feasible, more goal-faithful, and more diverse plans than prompt-based, supervised fine-tuning, and standard DPO baselines.