Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix It

The paper reports that chain‑of‑thought (CoT) supervised fine‑tuning (SFT) improves reasoning but systematically harms long‑context recall in hybrid linear‑attention models such as HypeNet and Jet‑Nemotron. Retrieval performance on the Needle‑In‑A‑Haystack benchmark drops dramatically after CoT‑SFT, especially with harder settings and longer contexts. The authors introduce QK‑Restore, a training‑free method that reinstates the query‑key projection matrices from the pre‑SFT checkpoint, which recovers long‑range recall while preserving reasoning gains.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• fine-tuning