Towards Feedback-to-Plan Decisions for Self-Evolving LLM Agents in CUDA Kernel Generation

Large language models (LLMs) have shown strong empirical gains as self-evolving agents for CUDA kernel generation, driven by feedback-conditioned planning across generations. However, how planning decisions attribute and combine heterogeneous feedback signals remains opaque. Standard end-to-end ablations fail to resolve this question, as iterative planning amplifies early perturbations and conflates feedback effects with trajectory-dependent drift. We introduce \texttt{CUDAnalyst}, a unified analysis layer for controlled, generation-level attribution of planning decisions to feedback components via trajectory freezing and selective feedback injection. \texttt{CUDAnalyst} enables stable generation-level evaluation and principled coalitional-style attribution of feedback effects and interactions. Our results show that explicit planning is beneficial only when feedback is aligned, that effective planning emerges from structured multi-feedback interactions, and that high-level plans from stronger reasoning models can partially transfer to weaker ones. These trends hold across reference backbones, representative workloads, and reference induction regimes, indicating that the identified feedback-to-plan structure is robust within the controlled axes studied.

Large Language Models (LLMs) are now capable of writing "CUDA kernels", the specialized code that powers high-performance computing. As AI agents, they improve by "self-evolving": writing code, receiving feedback from tools, and planning their next steps. However, this process is a "black box." Previously, we couldn't tell which feedback actually caused an improvement, as small changes early on create a "snowball effect" (trajectory drift), where tiny deviations lead to completely different outcomes, making it impossible to isolate what truly worked. To solve this, we developed CUDAnalyst. It acts like a controlled laboratory by "freezing" the AI at a specific moment to prevent these unpredictable shifts. By selectively showing the AI different feedback in a stable environment, we can precisely measure how each signal influences its plan. We found that effective planning requires highly accurate feedback and works best when multiple signals interact. Furthermore, "smart" plans from advanced models can be transferred to help simpler models perform better. These patterns remain consistent across different AI models and types of programming tasks, proving their reliability. This research makes AI programming more transparent and reliable for the future of computing hardware.