arXiv Machine Learning

Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory

The paper introduces Double-Take (DT), a memory architecture designed to improve capacity, binding, and selectivity in long-context processing. DT addresses two key failures in large-chunk test-time training: interference from single-step writes and the inability of self-associative writes to form cross-position bindings, by employing multi-step unpinned updates, cross-position mixing, and bidirectional change-sensitive writes. Evaluations on genomic data show DT achieving high recall and AUROC across increasing context sizes, and zero-shot transfer from human to mouse, demonstrating the importance of these three memory requirements.

arXiv Machine Learning
Oct 16

Long-Context Drafting Does Not Require Long-Context Drafters

The paper shows that long‑context drafting can be performed with short‑context drafters that are frozen and run beyond their training length, using two inference‑time mechanisms to keep their cost bounded. By bounding the attention cache and re‑basing rotary position indices, the authors demonstrate that the speedup achieved with a short‑context drafter persists up to two million tokens across multiple target‑drafter pairs and corpora. They prove that when a common index shift leaves the drafter’s computation unchanged, the acceptance gap can be tightly bounded, ensuring consistent performance.

By Aref Jafari, Vansh Bhatia, Matt Qin, Subhajit Dutta Chowdhury, Ben Lee, Vikram Appia, Ali Ghodsi, Mehdi Rezagholizadeh
arXiv Machine Learning
Oct 5

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

The paper introduces a Fisher-guided submodular data selection method for continual pre‑training of large language models, addressing the forgetting problem that arises when a target‑domain corpus overwrites pretrained knowledge. By decomposing candidate gradients into anchor and frontier components and optimizing a log‑determinant submodular objective, the method selects data that both improves target‑domain performance and limits forgetting. Experiments on TinyLlama‑1.1B and Llama‑3.1‑8B show that the selector achieves superior adaptation and forgetting control while using ten times fewer tokens than traditional replay strategies.

By Zhenghao Zhao, Gaowen Liu, Zhiling Lan, Yan Yan
arXiv AI
Jul 14

Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams

arXiv:2607. 10441v1 Announce Type: cross Abstract: Context engineering decides what information a model carries forward, and current designs meter it in tokens: compressing the past into a bounded recurrent state, keeping a key-value entry for every token, or imposing a fixed budget through a window or eviction rule.

By Siddharth Pal, Viktoria Rojkova
arXiv Machine Learning
Jul 10

What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

arXiv:2607. 08032v1 Announce Type: new Abstract: Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions.

By Ashwin Gerard Colaco, Nada Lahjouji