arXiv Machine Learning By Vakul Raghavan, Utkarsh Grover, Mingyang Mao, Chengwen Wang, Xiaomin Lin

Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory

Read the original on arXiv Machine Learning →

The paper introduces Double-Take (DT), a memory architecture designed to improve capacity, binding, and selectivity in long-context processing. DT addresses two key failures in large-chunk test-time training: interference from single-step writes and the inability of self-associative writes to form cross-position bindings, by employing multi-step unpinned updates, cross-position mixing, and bidirectional change-sensitive writes. Evaluations on genomic data show DT achieving high recall and AUROC across increasing context sizes, and zero-shot transfer from human to mouse, demonstrating the importance of these three memory requirements.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
Oct 16

Long-Context Drafting Does Not Require Long-Context Drafters

The paper shows that long‑context drafting can be performed with short‑context drafters that are frozen and run beyond their training length, using two inference‑time mechanisms to keep their cost bounded. By bounding the attention cache and re‑basing rotary position indices, the authors demonstrate that the speedup achieved with a short‑context drafter persists up to two million tokens across multiple target‑drafter pairs and corpora. They prove that when a common index shift leaves the drafter’s computation unchanged, the acceptance gap can be tightly bounded, ensuring consistent performance.

By Aref Jafari, Vansh Bhatia, Matt Qin, Subhajit Dutta Chowdhury, Ben Lee, Vikram Appia, Ali Ghodsi, Mehdi Rezagholizadeh