PADA-Coder: Improving Plan-Following Code Generation via Perturbation-Verified Attention Distillation and Dynamic Alignment

The Plan-then-Code paradigm effectively enhances Large Language Models (LLMs) in complex code generation by decomposing reasoning into explicit, interpretable steps. However, introducing the plan and verification report substantially enlarges the context, which in turn misdirects the model’s attention toward irrelevant tokens and the most recently generated code. This effect leads the model to overlook critical constraints and to generate incorrect code, especially for small-scale LLMs (less than 8B). To address this issue, we propose \textbf{P}erturbation-Verified \textbf{A}ttention \textbf{D}istillation and Dynamic \textbf{A}lignment (PADA). PADA identifies the key tokens most critical to the student model and constructs the optimal attention target matrix, dynamically aligning the student’s focus with key tokens for each plan step. We evaluate PADA with two teacher models and three student models across seven benchmarks, and the results show that PADA improves Pass@1 by up to 16.7\% and outperforms SOTA methods in overall average performance.

When asking AI to write complex code, we usually tell it to make a plan first. However, smaller AI models often "zone out" during the coding process. They get distracted by irrelevant information or fixate only on the code they just wrote, causing them to forget their initial plan and make mistakes. To fix this, we developed PADA, which acts like a "dynamic spotlight" for the AI. First, PADA tests and identifies the most critical words in the plan that determine if the code will be correct. Then, as the AI writes code step-by-step, PADA actively corrects the AI's focus, forcing it to concentrate on the specific plan step it is currently executing. This approach significantly fixes the AI's attention issues. Our tests show that small AI models using PADA improved their first-try coding accuracy by up to 16.7%. Impressively, this allows them to match or even beat massive, expensive top-tier AI systems on complex programming tasks. This research makes high-quality, affordable automated coding tools much more accessible.