QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture

The field of computer architecture, which bridges high-level software abstractions and low-level hardware implementations, remains absent from current large language model (LLM) evaluations. To this end, we present QuArch (pronounced ‘quark’), the first benchmark designed to facilitate the development and evaluation of LLM knowledge and reasoning capabilities specifically in computer architecture. QuArch v1.0 provides a comprehensive collection of 2,671 expert-validated question-answer (QA) pairs covering various aspects of computer architecture, including processor design, memory systems, and interconnection networks. Our evaluation reveals that while frontier models possess domain-specific knowledge, they struggle with skills that require higher-order thinking in computer architecture. Frontier model accuracies vary widely (from 34% to 73%) on these advanced questions, highlighting persistent gaps in architectural reasoning across analysis, design, and implementation QAs. Furthermore, via fine-tuning we find that QuArch can translate to improved performance on a realistic memory hierarchy design task, resulting in up to 1.99× more area-efficient solutions and up to 40% more viable solutions overall. By holistically assessing fundamental skills, QuArch provides a foundation for building and measuring LLM capabilities that can accelerate innovation in computing systems. The QuArch benchmark and leaderboard are publicly available at: https://quarch.ai/.

Large language models (LLMs) continue to accelerate work in fields like computer programming and have even driven advances in areas such as mathematics. Motivated by this recent progress, we ask how LLMs might help advance computer architecture, the field that studies how computing systems are designed and how hardware and software interact. To this end, we compiled the materials used to train real computer architects: academic exams and papers, expert-written questions, and industry documents. Using these sources, we created QuArch, the first question-answering benchmark for computer architecture. QuArch evaluates both an LLM’s knowledge of computer architecture concepts and its ability to reason about architecture-related problems. By evaluating QuArch across a broad range of both small and large models, we identify key areas where LLMs must improve before they can be reliably applied to computer architecture tasks. For example, we show that while modern LLMs already possess substantial factual knowledge about computer architecture, they struggle to use their knowledge to reason about complex architectural trade-offs and problems. QuArch will be freely available to the public to help continuously track LLM progress in this important domain.