arXiv Machine Learning

YANchor-4B: Effective Long-Horizon Reasoning in O(N) Time with O(1) Memory

YANchor-4B is a recurrent model that achieves long-horizon reasoning in linear time while using constant memory by preserving key information as anchors for later retrieval. It outperforms linear-time, constant-state baselines on challenging math benchmarks, achieving 82.93% mean pass@1 on AIME 2024–2026 and 63.64% on HMMT. The model also provides several-fold higher batched long-generation throughput than Transformer and hybrid baselines on H100, and demonstrates superior general-purpose capabilities across dozens of benchmarks.

arXiv AI

Aug 5

PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memory

arXiv:2608. 03048v1 Announce Type: cross Abstract: Long-context reasoning remains a critical bottleneck for large language models, as recent recurrent-memory approaches face two inherent challenges: sequential chunk-wise updates can overwrite early critical evidence with later irrelevant content, and serial inter-chunk dependencies limit parallelism and cause latency to increase with context length.

By Dawei Liu, Haixu Song, Shuang Cheng, Shijie Wang, Haozheng Hou, Kaifeng Liu, Ermo Hua, Zhonghang Yuan, Zhijie Zhong, Yuchen Fan, Biqing Qi, Bowen Zhou

llms nlp reinforcement-learning benchmarks

arXiv AI

Oct 15

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

arXiv:2610.15715v1 Announce Type: new Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...

By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber

llms rag efficiency

arXiv Computation and Language

Aug 24

Memory Augmentation Unlocks Efficient Chain-of-Thought Reasoning

arXiv:2608.21265v1 Announce Type: new Abstract: Large language models often rely on Chain-of-Thought (CoT) reasoning to solve complex tasks, but verbose reasoning traces introduce substantial inferen...

By Simeng Zhang, Yilong Chen, Wenyuan Zhang, Zhenyu Zhang, Yao Chen, Junyuan Shang, Tingwen Liu

llms nlp benchmarks

arXiv AI

Jul 29

Penelope: Localized Latent Recurrence for Efficient Structured Reasoning

arXiv:2607. 25915v1 Announce Type: new Abstract: Complex structured reasoning tasks often require additional computation, yet current language models obtain it mainly by increasing parameter scale or by serializing intermediate steps as chain-of-thought (CoT) tokens.

By Yutong Chen, Shouqian Shi, Xinran Liu, Haochen Wang, Jiaying Wang, Tianxing Xu, Yuanxi Wang, Zirui Ding

llms diffusion efficiency benchmarks

arXiv Machine Learning

Aug 14

MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

arXiv:2608. 12435v1 Announce Type: new Abstract: Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length.

By Ming Zhang, Kaisen Yang, Shu Yu, Ermo Hua, Ning Ding, Xia Hu, Bowen Zhou, Chaochao Lu, Youbang Sun

llms

arXiv Machine Learning

Sep 7

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

BeaconKV is a training‑free key‑value cache compression technique for Large Reasoning Models that uses beacon queries—compact representatives of query clusters—to predict which KV pairs will be revisited during long‑horizon reasoning. By focusing on Thought Revisiting Tokens that re‑attend distant context, BeaconKV reduces memory usage up to 5.8× and improves throughput by over 4.3× while largely preserving cache accuracy across multiple open‑source LRMs and reasoning benchmarks.

By Janghyeon Kim, Minsoo Kim, Kyuhong Shim, Jungwook Choi

rag efficiency benchmarks

arXiv AI

Oct 1

What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling

arXiv:2609.39967v1 Announce Type: cross Abstract: Recursive reasoning models apply a small shared Transformer block many times to refine a latent state. This gives them large effective depth with few...

By Yuliana Shakhvalieva, Dmitrii Kharchev, Viacheslav Bezrukov, Inessa Fedorova, Dmitry Bocharov, Ivan Oseledets, Valerii Ternovskii

llms

arXiv AI

Jun 26

CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

arXiv:2606. 27229v1 Announce Type: cross Abstract: Recurrent models must forget in order to remember, yet the state of the art decides what to erase without consulting what is stored -- the gate sees only the arriving token, not the memory it is about to modify.

By Sayak Dutta

llms efficiency benchmarks

arXiv Machine Learning

Aug 4

Structured Recurrent Mixers for Massively Parallelized Sequence Generation

arXiv:2605. 08696v4 Announce Type: replace-cross Abstract: Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that process sequence elements in parallel during training, which results in greater training efficiency and stability at the expense of lower inference throughput.

By Benjamin L. Badger

llms reinforcement-learning efficiency

arXiv AI

Aug 6

Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modeling

arXiv:2608. 02347v2 Announce Type: replace Abstract: Recurrent linear attention models (RLAs) such as Mamba offer efficient linear-time sequence modeling as an alternative to Transformers, yet their fixed-capacity recurrent states limit long-sequence modeling.

By Qinwen Wang, Jieping Luo, Aoxiang Qin, Ruoyu Zhao, Jianxiong Tang, Wei Zhang, Zhichao Lu, Luziwei Leng

llms

arXiv AI

Jul 14

MetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Models

arXiv:2603. 01331v3 Announce Type: replace-cross Abstract: Discrete diffusion language models (dLLMs) generate text by iteratively denoising a masked sequence.

By Kejing Xia, Mingzhe Li, Lixuan Wei, Zhenbang Du, Xiangchi Yuan, Dachuan Shi, Qirui Jin, Wenke Lee

llms diffusion benchmarks

arXiv Computation and Language

Oct 15

Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules

arXiv:2610.19210v1 Announce Type: new Abstract: Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference r...

By Sarah Pan, Ali Behrouz

llms benchmarks

Related stories