Probing Hybrid Language Models for In-Context Recall

Hybrid language models that interleave quadratic-attention with linear-attention layers can match quadratic-attention models on in-context recall while being more efficient at long context, but how they actually handle recall inside the architecture is not yet understood. Recent mechanistic work locates the recall circuit at the prediction position via head ablations on a few publicly-released hybrids, where attention placement is fixed by the original training recipe. We pretrain placement-controlled hybrids at 340M alongside quadratic-attention and linear-attention baselines. To see how they perform recall, we construct the sequence-layer map: a 2D probe-accuracy matrix over layers and token positions in the residual stream, applied uniformly across architectures. Attention placement determines whether a hybrid recalls, while standard language-modeling evaluations stay flat across placements. At the prediction position, the recall signal undergoes a sharp layer-localized phase transition only in hybrids able to recall and in quadratic-attention models. Across the sequence, quadratic-attention models drop the recall signal and look it up at the prediction position; hybrids inherit this pattern depending on where the attention layer sits. A head-level analysis at 340M and on OLMo-Hybrid 7B further localizes the phase transition to a small set of attention heads, including some we can only see by reading what each head writes into the answer-token logits. Together, these results give a controlled view of how pretrained hybrid language models handle in-context recall, and why some succeed where others fail.