← Back to all news
arXiv Computation and Language By Yunao Zheng, Bin Wen, Xiaojie Wang, Kaiyu Jiang, Xuanyu Zheng, Changyi Liu, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Jiankang Chen, Xu Wang, Tingting Gao, Han Li

GSM: Efficient Language Modeling with Shared Global State

Read the original on arXiv Computation and Language →

The paper introduces the Global State Model (GSM), a causal encoder-decoder architecture that separates historical information acquisition from deep computation. GSM’s encoder aggregates long-range information via multi-stage sparse retrieval into a fixed-size shared state, while the decoder transforms this state without repeated historical access, making attention and cache costs independent of history length. Experiments show GSM matches or exceeds the performance of DeepSeek-V4.1-Flash baselines across scales, while improving training and inference throughput and reducing cache overhead.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 15

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

arXiv:2610.15715v1 Announce Type: new Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...

By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber
llmsragefficiency
More like this →
Hugging Face Trending Papers
Sep 17

On-Demand Attention: Language Models Know When to Recall

The paper introduces On‑Demand Attention (ODA), a local‑first decoding strategy that predicts when a pretrained language model would benefit from global attention. By training only a lightweight recall head, ODA selectively triggers global attention during generation, keeping pretrained weights unchanged and preserving the full key‑value cache for future recall. Experiments on Qwen, Gemma, and hybrid‑attention models show that ODA largely recovers performance lost with local attention while significantly cutting global reads, enabling faster long‑context inference.

llmsagentsefficiency
More like this →
arXiv Computation and Language
Sep 18

On-Demand Attention: Language Models Know When to Recall

The paper introduces On‑Demand Attention (ODA), a decoding strategy that lets pretrained language models decide when to use global attention based on a lightweight recall head. ODA keeps the original model weights unchanged, only training the recall head, and can be implemented with GPU‑side conditional execution to reduce global reads. Experiments on Qwen, Gemma, and hybrid‑attention models show that ODA largely recovers performance lost by local attention while cutting the number of global attention operations.

By Haibo Feng, Ruiqi Liang, Hanyang Peng, Shiqi Yu
llmsagentsefficiency
More like this →