Probe Choice Changes Canary-Memorization Verdicts: Three Post-Hoc Disagreement Case Studies in a Text-Dominant LoRA-Tuned Autoregressive Testbed

Distinguishing memorization from generalization in autoregressive generative models requires probes whose verdicts are stable under the choices of which tokens to score and which scoring rule to use. A common family averages teacher-forced negative log-likelihood (NLL) over a fixed prefix window of length K. We audit the internally pre-specified K = 20 window in a single Qwen2.5-VL-7B testbed and report three post-hoc cases where this probe disagrees with full-span secret NLL or with greedy exact-recall: (C3) damage lands on hex tokens outside the K = 20 window — the probe stays flat while hit@1 drops; (C4) the probe moves, but almost all of the movement sits on non-secret preamble while the secret span and hit@1 are unchanged; (C5) the probe drops on an undertrained baseline while the full-span hex estimate is positive and hit@1 = 0. Each case is a distinct failure mode of truncated prefix-NLL: false negative under window truncation, false positive from non-secret span drift, and an ambiguous in-window drop that only a paired decoy audit can resolve. We recommend reporting (i) full-span secret NLL, (ii) a span-localised decomposition, (iii) behavioural exact-recall at k ≥ 4, and (iv) decoy probes before asserting secret-specificity. All evidence concerns controlled canary targets, not natural training-example memorization; a smaller-family stress test surfaces additional disagreements but does not replicate every 7B mechanism, and magnitudes remain testbed-specific.