← Back to all news
arXiv AI By John Langford, Nathan Godey, Giovanni Monea, Yoav Artzi, Harry Dong, Ying Fan, Gustavo de Rosa, Zheng Zhan

Free Pause Tokens

Read the original on arXiv AI →

The paper introduces the concept of a free pause token, which allows a language model to perform additional computation for each next-token prediction without increasing the sequence length or KV cache usage. This pause token operates as a parallel prediction stream over a shared backbone, improving next-token accuracy by 2–3 centi‑nats on a 1‑billion‑parameter model. While it adds minimal inference overhead, the primary cost lies in training, where compute requirements rise modestly (up to 1.14×) yet most benefits are retained.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 7

Almost Free State Prediction Separation

The paper introduces a free pause token that allows a language model to perform an extra parallel prediction stream without adding to the sequence length. This technique improves next‑token accuracy by 2–3 centi‑nats on a 1B‑parameter model while adding negligible inference latency or context overhead. Training costs increase modestly, with compute overhead reduced to about 1.14× compared to an optimized pretraining pipeline, preserving most benefits.

By John Langford, Nathan Godey, Giovanni Monea, Yoav Artzi, Harry Dong, Ying Fan, Gustavo de Rosa, Zheng Zhan
llmsefficiency
More like this →
arXiv AI
Jul 17

Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs

arXiv:2607. 14107v1 Announce Type: cross Abstract: The inference efficiency of diffusion large language models (dLLMs) is constrained by two challenges: bidirectional attention precludes efficient KV-cache reuse, while increasing decoding parallelism with static confidence thresholds can compromise generation quality.

By Mingyu Lee, Akshat Ramachandran, Souvik Kundu, Tushar Krishna
llmsdiffusionefficiencybenchmarks
More like this →
arXiv Machine Learning
Jul 20

DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts

arXiv:2605. 15422v3 Announce Type: replace Abstract: Modern RL post-training methods such as GRPO and DAPO train on N response sequences of R tokens sampled from a shared prompt of P tokens, but standard FlashAttention replicates all P prompt tokens N times across both forward and backward passes -- duplicating compute and memory on identical hidden states.

By Jiading Gai, Shuai Zhang, Xiang Song, Bernie Wang, George Karypis
reinforcement-learningefficiency
More like this →