Structured Progressive Knowledge Activation for LLM-Driven Neural Architecture Search

This paper focuses on a key challenge in Neural Architecture Search (NAS): integrating established architectural knowledge while exploring new designs under expensive evaluations. Large language models (LLMs) are a promising assistant for NAS because they can translate rich architectural and coding priors into executable code edits. However, in practice, seemingly local revisions often propagate into non-local behavioral and performance shifts because a single edit can inadvertently couple multiple interacting functional factors, a phenomenon we refer to as functional entanglement. To make LLM knowledge usable under such entanglement, we propose Structured Progressive Knowledge Activation (SPARK), which activates relevant priors by explicitly selecting the functional factor to modify and conditioning the edit on that factor. This factor-conditioned editing reduces entangled side effects and yields more targeted, reliable architecture modifications. On CLRS-DFS, SPARK achieves a 28.1x sample-efficient architecture evolution speedup and yields a 22.9% relative improvement in OOD accuracy. Our code is available at https://github.com/AIM-ResearchLab/SPARK.

Designing new AI systems is a complex and time-consuming process. While developers can use advanced AI assistants to suggest code improvements, there is a major catch: a small tweak in one part of the code often accidentally breaks or alters other parts of the system. To solve this, researchers developed a new method called SPARK. Instead of letting the AI make broad, general edits, SPARK forces it to focus on changing just one specific function at a time. This targeted approach prevents messy, unexpected side effects. As a result, SPARK makes designing new AI models significantly faster, more reliable, and much more accurate—all without requiring any extra computing power.