SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks

We consider the task of end-to-end spreadsheet generation, where language models are prompted to produce spreadsheet artifacts to satisfy users' explicit and implicit constraints, specified in natural language. We introduce SpreadsheetArena, a platform for evaluating models' performance on the task via blind pairwise evaluations of LLM-generated spreadsheet workbooks. As with other complex, open-ended tasks, relevant evaluation criteria can vary greatly across use cases and prompts, often in ways that are difficult to formalize. Compared to general dialogue or text generation settings, spreadsheet generation presents unique challenges and opportunities: the task output structure is well-defined and multi-dimensional, and there are often complex interactivity and layout considerations. We observe that stylistic, structural, and functional features of preferred spreadsheets vary meaningfully across prompts. Expert evaluations of spreadsheets for finance prompts suggest that even highly ranked models do not reliably produce spreadsheets aligned with domain-specific best practices. We host a live arena and release a dataset of prompts, generated spreadsheets, and preference votes, which we hope will facilitate further study of tasks operating over spreadsheets as a challenging and interesting class of complex, open-ended tasks for LLMs.

Hundreds of millions of professionals around the world use spreadsheets for work tasks, many of which are difficult and time-consuming. Meanwhile, artificial intelligence (AI) systems' capabilities on spreadsheet tasks is still relatively underexplored and underdeveloped compared to some other types of tasks like software engineering. In response, we built a preference arena for open-ended spreadsheet generation by large language models, where users write and submit their spreadsheet generation requests and vote on paired battles between anonymized models' candidate generations. The arena and its live results are useful for comparing today's most capable AI models with each other. Additionally, preference votes on challenging tasks like these can be useful as a dataset for training other AI models to perform well on similar tasks.