Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesis

Deploying Large Language Models to data-scarce programming domains poses significant challenges, particularly for kernel synthesis on emerging Domain-Specific Architectures where a ``Data Wall'' limits available training data. While models excel on data-rich platforms like CUDA, they suffer catastrophic performance drops on data-scarce ecosystems such as NPU programming. To overcome this cold-start barrier without expensive fine-tuning, we introduce EvoKernel, a self-evolving agentic framework that automates the lifecycle of kernel synthesis from initial drafting to continual refining. EvoKernel addresses this by formulating the synthesis process as a memory-based reinforcement learning task. Through a novel value-driven retrieval mechanism, it learns stage-specific Q-values that prioritize experiences based on their contribution to the current objective—whether bootstrapping a feasible draft or iteratively refining latency. Furthermore, by enabling cross-task memory sharing, the agent generalizes insights from simple to complex operators. By building an NPU variant of KernelBench and evaluating on it, EvoKernel improves frontier models' correctness from 11.0\% to 83.0\% and achieves a median speedup of 3.60$\times$ over initial drafts through iterative refinement. This demonstrates that value-driven experience accumulation allows general-purpose models to master the kernel synthesis task on niche hardware ecosystems.

Today's AI models write impressive code for widely used platforms because billions of code examples exist online to learn from. Many emerging computing domains, however, are starved of such data — new specialized hardware architectures keep appearing, but documentation and example code remain scarce. Even the most capable models get such code right only about one time in ten on these less-supported platforms. We built EvoKernel, a system that turns a general-purpose AI model into an effective programmer for such a data-scarce domain, without retraining the model itself. The idea is to give the model a growing notebook of its own past attempts: each time it tries a problem, the system records what worked and learns which past notes help most on the next problem. Lessons from easier problems help the model tackle harder ones. Using the Ascend architecture as a representative example of an emerging, low-resource domain, EvoKernel raises the AI's success rate from 11% to 83% and makes solutions roughly 3.6 times faster than the first correct attempt. The same approach should help AI models quickly become competent on other emerging hardware where training data is limited.