Beyond Deduplication: Sharp Retrieval Limits for Attention with Correlated Distractors

The paper studies how redundant context influences attention mechanisms, distinguishing between normalization effects from repeated items and score variability within correlated groups. Using a planted retrieval model with exponentially many Gaussian distractors organized into large clusters, the authors derive exact retrieval thresholds via a two‑level generalized random energy model and Gaussian interpolation. They show that subtracting the log cluster size from pre‑softmax scores removes multiplicity but fails to eliminate extreme within‑cluster scores, establishing a sharp limit on signal strength that cannot be overcome by temperature scaling alone; averaging scores before exponentiation mitigates this effect but only when the relevant item is a singleton.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• rag