arXiv Computation and Language

Reinforced Fast Weights via Next-Sequence Prediction

The paper "Reinforced Fast Weights via Next-Sequence Prediction" introduces ReFINE, a method that extends the next-token prediction objective to next-sequence prediction for fast weight architectures. ReFINE selects informative token positions, generates multi-token rollouts, assigns sequence-level rewards, and optimizes the model using Group Relative Policy Optimization. Experiments on LaCT-760M and DeltaNet-1.3B demonstrate that ReFINE consistently outperforms standard next-token prediction fine‑tuning across retrieval, long‑context question answering, and LongBench tasks.

arXiv Computation and Language

Oct 15

Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules

arXiv:2610.19210v1 Announce Type: new Abstract: Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference r...

By Sarah Pan, Ali Behrouz

llms benchmarks

arXiv Machine Learning

Aug 4

Learning What to Remember: Test-Time Training via Context Distillation

arXiv:2608. 01672v1 Announce Type: cross Abstract: Effective long-context modeling is not merely about retaining more of the past, but about preserving the information that may prove relevant later.

By Zixuan Wang, Xingyu Dang, Rui-Jie Zhu, Zixin Wen, Hengyu Fu, Wenhao Chai, Jason D. Lee

llms efficiency

arXiv AI

Jul 9

Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalization

arXiv:2604. 01577v3 Announce Type: replace-cross Abstract: We study out of distribution generalization in streaming tasks where models are trained on short sequences but must operate over much longer, unknown horizons under bounded memory.

By Shota Takashiro, Masanori Koyama, Takeru Miyato, Yusuke Iwasawa, Yutaka Matsuo, Kohei Hayashi

llms reinforcement-learning safety

arXiv Machine Learning

Jun 29

Learning to Evict from Key-Value Cache

arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.

By Luca Moschella, Laura Manduchi, Ozan Sener

llms agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jun 5

Pretraining Recurrent Networks without Recurrence

arXiv:2606. 06479v1 Announce Type: new Abstract: Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations.

By Akarsh Kumar, Phillip Isola

llms

arXiv Machine Learning

Sep 2

Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior

arXiv:2605.26797v2 Announce Type: replace Abstract: We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden...

By Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, Yelong Shen

llms

arXiv Machine Learning

Aug 14

MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

arXiv:2608. 12435v1 Announce Type: new Abstract: Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length.

By Ming Zhang, Kaisen Yang, Shu Yu, Ermo Hua, Ning Ding, Xia Hu, Bowen Zhou, Chaochao Lu, Youbang Sun

llms

arXiv Computation and Language

Sep 1

Test-Time Training with Next-Token Prediction

arXiv:2606.21803v2 Announce Type: replace Abstract: Next-token prediction is the self-supervised signal that trains language models, and every observed prompt token provides the same signal at test t...

By Xuan Ouyang, Zefan Cai, Junjie Hu

llms benchmarks

arXiv Machine Learning

Jun 16

Next-Latent Prediction Transformers Learn Compact World Models

arXiv:2511. 05963v4 Announce Type: replace Abstract: Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens.

By Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, John Langford

llms diffusion benchmarks safety

arXiv AI

Oct 15

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

arXiv:2610.15715v1 Announce Type: new Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...

By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber

llms rag efficiency

arXiv AI

Jun 9

End-to-End Context Compression at Scale

arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.

By Ang Li, Sean McLeish, Haozhe Chen, Nimit Kalra, Zaiqian Chen, Artem Gazizov, Venkata Anoop Suhas Kumar Morisetty, Bhavya Kailkhura, Harshitha Menon, Zhuang Liu, Brian R. Bartoldson, Tom Goldstein, Sanae Lotfi, Micah Goldblum, Pavel Izmailov

llms rag agents efficiency

arXiv AI

Aug 18

Proteus: Incremental Memory Activation for Long-Context Sequence Modeling

arXiv:2608. 16844v1 Announce Type: cross Abstract: The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state.

By Reza Bayat, Ali Behrouz, Vahab Mirrokni, Aaron Courville

benchmarks

Related stories