Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries
Abstract
Unlearning evaluations for reasoning models often split a generation into a reasoning trace and an answer, then treat cases where the answer no longer contains a forgotten canary but the trace still does as evidence of residual memorization in the weights. We audit this interpretation. We LoRA-memorize 60 fictional author bios with distinctive canaries into DeepSeek-R1-Distill-Qwen-7B, unlearn them with answer-masked NPO, and evaluate head-conditioned continuation under a parser-split canary-containment metric. The standard metric produces a bypass pattern: after NPO, answer-side containment drops from 1.00 to 0.60 while thinking-trace leakage remains 0.83, giving a gap of +0.23. However, on the same weights, replacing the autoregressive trace with short non-canary prefills changes answer-side continuation by a comparable amount; on a second seed, the direction of this intervention reverses. A matched teacher-forced continuation probe remains high across prefills, and on DeepSeek-R1-Distill-Llama-8B the same parser metric flips sign under format drift because the closing tag is missing. These results do not prove that forgotten information is absent. Rather, they show that parser-split reasoning-trace leakage is non-identifying: positive trace-answer gaps can conflate residual memorization with decode-time prefix sensitivity and parser conventions. We recommend fixed-prefill template swaps and teacher-forced continuation checks as cheap sanity checks for memorization and unlearning audits on reasoning foundation models.