InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information

Competitive programming is increasingly being used to evaluate the algorithmic reasoning capabilities of large language models (LLMs). However, existing benchmarks primarily focus on full-information tasks where all problem inputs are provided upfront. This overlooks a critical dimension of algorithmic reasoning: the ability of generated programs to operate when key information is not revealed upfront. Interactive problems, a distinctive component of competitive programming, embody this challenge. These problems require programs to engage in multi-round interaction with an interactor (a judge program) under strict protocol constraints and limited query budgets, with new information revealed only in response to queries. To address this gap, we introduce InteractBench, a benchmark comprising 322 high-quality interactive problems curated from Codeforces, AtCoder, IOI, and ICPC. Each problem is packaged with executable local interactors, enabling fully offline evaluation. Unlike existing benchmarks, InteractBench assesses whether model-generated code can acquire information and track state dynamically. Our evaluation reveals a significant interaction gap: even the most advanced reasoning models achieve limited success on interactive problems. Beyond success rates, we propose a fine-grained failure taxonomy to diagnose the root causes of these deficiencies. Although algorithmic logic errors remain dominant, protocol violations and query-budget overruns are frequent. Code is available at https://github.com/kmsgk0/InteractBench.

Competitive programming is often used to test whether large language models can reason about algorithms and write correct code. However, most existing benchmarks assume that all input information is available from the beginning, and do not test whether a program can discover hidden information through multi-round interaction with an interactor (a judge program). We introduce InteractBench, a benchmark of 322 interactive competitive-programming problems from Codeforces, AtCoder, IOI, and ICPC. Each problem includes a local executable interactor, so model-generated programs can be evaluated offline. Unlike standard coding benchmarks, InteractBench tests whether models can ask useful queries, follow the strict interaction protocol, stay within query limits, and keep track of changing information during execution. Our experiments show that even strong reasoning models have limited success on these tasks. Many failures are still due to wrong algorithms, but protocol mistakes and excessive queries are also common. This suggests that current language models struggle with interactive problem-solving under hidden information.