UniCode: Augmenting Evaluation for Code Reasoning

Current coding benchmarks often overstate Large Language Model (LLM) capabilities due to static paradigms and data contamination, allowing models to exploit statistical shortcuts rather than genuine reasoning. To address this, we introduce UniCode, a generative evaluation framework that systematically probes LLM reasoning boundaries via: (1) multi-dimensional augmentation operators to create diverse algorithmic variants; (2) a scalable test generation pipeline achieving 94.5\% correctness without human-written solutions; and (3) fine-grained diagnostic metrics for rich error signals. Our evaluation of state-of-the-art models reveals a significant 31.2\% performance collapse. Critically, we observe a high variance across different reasoning axes, revealing a profound fragility under structural shifts despite surface-level robustness. Furthermore, we identify a ``seed-problem regression," where models fail by defaulting to memorized seed logic and inefficient complexities. Our evaluation code is publicly available at \url{https://github.com/grandsmile/UniCode}.

Why do AI models that seem like master coders in testing often fail in the real world? Currently, standard AI coding tests use fixed, predictable patterns. This allows Large Language Models (LLMs) to take "shortcuts" by memorizing past programming data rather than genuinely understanding computational logic. To expose this illusion of competence, we developed UniCode, an automated framework that systematically stress-tests AIs by transforming familiar coding tasks into fresh, un-memorizable variations. These include shifting the background story, modifying subtle operational rules, scaling up input data to demand highly efficient code, or blending entirely separate algorithmic concepts together. When we evaluated 19 state-of-the-art AI models using UniCode, their coding performance collapsed by an average of 31.2%. Crucially, our deep-dive diagnostics uncovered a striking behavioral flaw called "seed-problem regression": when faced with new, modified instructions, the AIs repeatedly stopped reasoning and blindly reverted back to the memorized logic of the original training problems. By dismantling these blind spots, UniCode provides a much-needed, robust diagnostic tool to help researchers build next-generation AI coding agents that truly reason rather than recite.