← Back to all news
arXiv Machine Learning By Samuel Larson

Constant-Memory Recall: Learned Associations in a Fixed Matrix State

Read the original on arXiv Machine Learning →

The paper introduces a small DeltaNet variant that uses a fixed-size recurrent memory to store 32 key‑value pairings per sequence. With only 32 KiB of matrix state, the model attains 99.95 % mean accuracy across three training seeds when selecting the correct value, and this performance persists even when the pre‑query context is extended to 1,798 tokens. Zeroing the first memory block eliminates recall, and exploratory tests with 48 pairs show near‑chance performance after a quarter of the training budget, indicating no clear capacity limit. Baseline models matched in parameters, including a Transformer, remain near chance, preventing a direct memory‑efficiency comparison.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

Hugging Face Trending Papers
Jul 2

A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--value associations compete, earlier facts are overwritten and needle recall degrades. Inspired by Complementary Learning Systems, we give linear attention a hippocampal complement.

llmsefficiency
More like this →
arXiv Machine Learning
Oct 5

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

The paper introduces a Fisher-guided submodular data selection method for continual pre‑training of large language models, addressing the forgetting problem that arises when a target‑domain corpus overwrites pretrained knowledge. By decomposing candidate gradients into anchor and frontier components and optimizing a log‑determinant submodular objective, the method selects data that both improves target‑domain performance and limits forgetting. Experiments on TinyLlama‑1.1B and Llama‑3.1‑8B show that the selector achieves superior adaptation and forgetting control while using ten times fewer tokens than traditional replay strategies.

By Zhenghao Zhao, Gaowen Liu, Zhiling Lan, Yan Yan
llmsbenchmarks
More like this →