Poster Wed, Jul 8, 2026 • 10:30 PM – 12:15 AM PDT HALL A #502

FrontierCS: Evolving Challenges for Evolving Intelligence

Qiuyang Mang ⋅ Wenhao Chai ⋅ Zhifei Li ⋅ Huanzhi Mao ⋅ Shang Zhou ⋅ Alexander Du ⋅ Hanchen Li ⋅ Shu Liu ⋅ Edwin Chen ⋅ Yichuan Wang ⋅ Xieting Chu ⋅ Zerui Cheng ⋅ Yuan Xu ⋅ Tian Xia ⋅ Zirui Wang ⋅ Tianneng Shi ⋅ Jianzhu Yao ⋅ Yilong Zhao ⋅ Qizheng Zhang ⋅ Charlie Ruan ⋅ Zeyu Shen ⋅ Kaiyuan Liu ⋅ Zhaoyang Hong ⋅ Alex Gu ⋅ Ziyi Zhang ⋅ Runyuan He ⋅ Dong Xing ⋅ Zerui Li ⋅ Zirong Zeng ⋅ Yige Jiang ⋅ Lufeng Cheng ⋅ Ziyi Zhao ⋅ Youran Sun ⋅ Suyang Zhong ⋅ Junpeng Wang ⋅ Donglin Li ⋅ Wenyuan Huang ⋅ Jialiang Gu ⋅ Wesley Zheng ⋅ Wangmeiyu Zhang ⋅ Ruyi Ji ⋅ Xuechang Tu ⋅ Zihan Zheng ⋅ Zhaozi Wang ⋅ Zexing Chen ⋅ Jingbang Chen ⋅ JIALU ZHANG ⋅ Aleksandra Korolova ⋅ Peter Henderson ⋅ Pramod Viswanath ⋅ Vijay Ganesh ⋅ Saining Xie ⋅ Zhuang Liu ⋅ Dawn Song ⋅ Sewon Min ⋅ Ion Stoica ⋅ Joseph E Gonzalez ⋅ Jingbo Shang ⋅ Alvin Cheung

Project Page

Abstract

We introduce FrontierCS, a benchmark of 240 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competitive programming participants and problem setters. Unlike existing benchmarks that focus on tasks with known optimal solutions, FrontierCS targets problems where the optimal solution is unknown, but the quality of a solution can be objectively evaluated. Models solve these tasks by implementing executable programs rather than outputting a direct answer. FrontierCS includes algorithmic problems, which are often NP-hard variants of competitive programming problems with objective partial scoring, and research problems with the same property. For each problem, we provide an expert reference solution and an automatic evaluator. Combining open-ended design, measurable progress, and expert curation, FrontierCS provides a benchmark at the frontier of computer-science difficulty. Empirically, we find that frontier reasoning models still lag far behind human experts, and that simply increasing reasoning budgets does not close this gap on open-ended challenges. Moreover, these models struggle to identify internal equivalence classes, and existing agentic frameworks also exhibit brittleness on such problems due to overfitting. FrontierCS thus offers a new lens into model capabilities on real frontier computer science problems.

Lay Summary

FrontierCS is a new benchmark for testing whether advanced AI systems can solve difficult, open-ended computer science problems. Many existing coding benchmarks ask models to produce one correct answer, which is then judged as simply right or wrong. In real research and engineering, however, problems often do not have a known best answer. Instead, many solutions may be valid, but some are much better than others. FrontierCS contains 240 expert-designed problems across algorithms and real computer science research areas. Each problem asks an AI model to write a working program, and the program is automatically checked and scored based on how good its solution is. For example, a model may need to pack shapes into the smallest possible space, reduce the cost of a system, or improve the speed of a computational method. These tasks allow partial progress to be measured, rather than only pass or fail. Our results show that even the strongest current AI models remain far behind human experts on these open-ended challenges. We also find that simply giving models more time to “think” does not reliably solve the problem, and that models can be sensitive to small changes in how a problem is written. FrontierCS therefore provides a more realistic way to measure progress toward AI systems that can help with challenging research and engineering work.