Detection Without Suppression: A Sign Flip in IOI Circuit Formation

Transformers trained on language modeling develop the ability to identify indirect objects (IOI) non-monotonically: accuracy drops below chance before recovering. We observe this dip across 15 training runs spanning two model families, three scales, and nine seed/data/initialization variants. Linear probes show that name-duplication information is linearly decodable at 99% accuracy at the S2 position on Pythia-160M during the dip, while IOI accuracy is 41%; a roughly 89% S2 probe baseline is already present at random initialization. Activation patching reveals a sign flip: replacing S2 with a non-duplicate control reduces the S-bias during the dip (∆LD = +0.94) but degrades performance after recovery (∆LD = −4.13), while the same intervention at a non-S2 position produces ∆LD = 0.000. Head ablation shows that S-inhibition heads, which eventually contribute 10.8× more to the logit difference than name movers, have not yet acquired large measurable effects during the below-chance phase.