arXiv AI

Free Pause Tokens

The paper introduces the concept of a free pause token, which allows a language model to perform additional computation for each next-token prediction without increasing the sequence length or KV cache usage. This pause token operates as a parallel prediction stream over a shared backbone, improving next-token accuracy by 2–3 centi‑nats on a 1‑billion‑parameter model. While it adds minimal inference overhead, the primary cost lies in training, where compute requirements rise modestly (up to 1.14×) yet most benefits are retained.

arXiv AI

Sep 7

Almost Free State Prediction Separation

The paper introduces a free pause token that allows a language model to perform an extra parallel prediction stream without adding to the sequence length. This technique improves next‑token accuracy by 2–3 centi‑nats on a 1B‑parameter model while adding negligible inference latency or context overhead. Training costs increase modestly, with compute overhead reduced to about 1.14× compared to an optimized pretraining pipeline, preserving most benefits.

By John Langford, Nathan Godey, Giovanni Monea, Yoav Artzi, Harry Dong, Ying Fan, Gustavo de Rosa, Zheng Zhan

llms efficiency

arXiv Machine Learning

Aug 24

Rethinking Expressivity and Efficiency in Test-Time Training

arXiv:2608.21308v1 Announce Type: new Abstract: Test-Time Training (TTT) enables long-context processing via continuous weight updates during inference, but current methods struggle to balance the ex...

By Zeyun Zhong, Joya Chen, Manuel Martin, Frederik Diederichs, Juergen Gall, Juergen Beyerer

efficiency

arXiv AI

Oct 2

Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads

arXiv:2610.00888v1 Announce Type: cross Abstract: Multi-token prediction (MTP) improves the throughput of autoregressive generation by enabling the language model to draft multiple next tokens per fo...

By Prachi Badarayani, Aidan Jay, Chenghui Zhou, Dayquan Julienne, Yuan Gao, Tianwei Chen, George Zerveas, Ishmam Zabir, Xiren Zhou, Chris Quirk, Xia Song

llms efficiency benchmarks

arXiv AI

Jul 17

Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers

arXiv:2607. 14427v1 Announce Type: new Abstract: A depth-recurrent transformer applies a weight-tied core a variable number of times, and prior work has shown that training with a randomized recursion count yields one checkpoint usable across a range of inference depths.

By Joe Logan

llms

arXiv AI

Jul 17

Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs

arXiv:2607. 14107v1 Announce Type: cross Abstract: The inference efficiency of diffusion large language models (dLLMs) is constrained by two challenges: bidirectional attention precludes efficient KV-cache reuse, while increasing decoding parallelism with static confidence thresholds can compromise generation quality.

By Mingyu Lee, Akshat Ramachandran, Souvik Kundu, Tushar Krishna

llms diffusion efficiency benchmarks

arXiv Machine Learning

Jul 20

DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts

arXiv:2605. 15422v3 Announce Type: replace Abstract: Modern RL post-training methods such as GRPO and DAPO train on N response sequences of R tokens sampled from a shared prompt of P tokens, but standard FlashAttention replicates all P prompt tokens N times across both forward and backward passes -- duplicating compute and memory on identical hidden states.

By Jiading Gai, Shuai Zhang, Xiang Song, Bernie Wang, George Karypis

reinforcement-learning efficiency

arXiv Machine Learning

Jul 13

HALO: Hybrid Adaptive Latent Reasoning for Language Models

arXiv:2607. 08775v1 Announce Type: cross Abstract: We study how to improve a frozen pretrained language model with a small amount of adaptive extra computation.

By Micah Zhang

llms benchmarks

arXiv AI

Sep 30

TokenCast: Forecasting Token Consumption During LLM Agent Execution

arXiv:2609.35760v2 Announce Type: replace-cross Abstract: When a large language model (LLM) agent executes the same task, token consumption can vary by over an order of magnitude across runs. The age...

By Chaoqian Ouyang, Ling Yue, Libin Zheng, Hanghui Guo, Shengxiang Xu, YiShu Wang, Ran Li, Jian Yin, Shaowu Pan, Shimin Di

llms agents

arXiv Computer Vision

Sep 15

StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models

arXiv:2609.13804v1 Announce Type: new Abstract: Visual prefixes account for a major portion of the per-layer computation in multimodal large language models (MLLMs), making visual-token pruning a dir...

By Hansen Zhang, Landi He, Mingde Yao, Lijian Xu

llms efficiency multimodal

arXiv AI

Jul 21

FlowBlock: Wavefront-Parallel Decoding for Self-Correcting Diffusion Language Models

arXiv:2607. 17652v1 Announce Type: new Abstract: Block-wise diffusion large language models (dLLMs) decode sequentially at the block level, enabling effective KV-cache reuse across blocks but making inter-block decoding strictly serial.

By Bing Tian, Haikun Liu, Xiaocheng Zhong, Zhuohui Duan, Zhaokai Luo, Huayi Jin, Zhiyong Wang, Xiaofei Liao

llms diffusion efficiency benchmarks

arXiv Machine Learning

Aug 11

Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching

arXiv:2608. 09444v1 Announce Type: new Abstract: A main promise of looped language models (LMs) is depth-adaptive inference.

By Kristian Schwethelm, Daniel Rueckert, Georgios Kaissis

llms rag efficiency

arXiv AI

Aug 24

MentorPulse: Refreshing Cross-Model Latent Guidance for Long-Form Generation

MentorPulse introduces a method for refreshing cross‑model latent guidance during long‑form generation. By compressing mentor states into a capped slot memory and updating it incrementally as new tokens are generated, the approach keeps the guidance signal fresh without resetting the student’s KV cache. Across thirteen datasets, MentorPulse closes more than half of the mentor‑student performance gap, outperforming existing techniques such as C2C, T2T, and LoRA, especially on long outputs.

By Ziwu Liu, Guozhong Li, Chen Qiu, Weiyang Kong, Panos Kalnis

fine-tuning efficiency

Related stories