arXiv Machine Learning

GDN Tree-Scan: Served Tree Verification for Recurrent-Hybrid Language Models

arXiv AI
Aug 24

TreeWY: Speculative Verification for Gated DeltaNet Hybrids

TreeWY introduces a speculative verification method for Gated DeltaNet (GDN) hybrid models that eliminates the need for per-draft-state snapshots. By applying a tree‑structured WY transform to the gated delta rule, each draft node’s output is computed with a single triangular solve, and only the accepted state is reconstructed on commit. Benchmarks on Qwen3.5 35B and 397B show reduced memory pressure, higher throughput, and lower time‑to‑first‑token in memory‑bound scenarios, while enabling wider, higher‑acceptance draft trees.

By Sneha Murthy Ghantasala
arXiv Machine Learning
Aug 4

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

arXiv:2608. 01651v1 Announce Type: cross Abstract: Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound.

By Li Wang, Yi Su, Xiabao Wu, Chiran You, Yongchao Liu, Zhan Qiu, Juelu Zhang, Jiajun Zheng, Fangxin Liu, Jie Zhang, Chen Tian, Chengying Huan
arXiv Computation and Language
Sep 23

LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay

The paper introduces LatentPort, a method that allows a language model to transfer its live memory to another model without requiring the receiver to reread the context. Experiments on a Qwen3.5 4B-to-9B sibling pair show that adding a Gated DeltaNet (GDN) persistent-state package reduces negative log‑likelihood by 0.747 nats/token and improves performance across 64 PG19 documents. The study also demonstrates that direct recurrent and convolution reuse outperforms learned GDN maps, and a 434,176‑parameter correction further narrows the performance gap to the native 9B model.

By Simon P. Villani
arXiv Computation and Language
Oct 19

DominoTree: Conditional Tree-Structured Drafting with Domino for Speculative Decoding

DominoTree is a speculative decoding method that builds a conditional tree of draft tokens using Domino’s path‑dependent causal correction while restricting to the top‑M candidates. It achieves significant speedups over standard autoregressive decoding—averaging 5.33× on Qwen3‑4B and 6.89× on Qwen3‑8B at temperature zero—and outperforms both Domino and DDTree variants across a range of temperatures and batch sizes. The approach also improves accepted token length and maintains higher acceptance rates than chain‑based methods in various serving scenarios.

By Saw S. Lin (Zhiqi Zhang), Jyh-Shing Roger Jang
arXiv AI
Oct 9

Where Draft Trees Lose Target Mass: Exit-Guided Speculative Decoding

The paper introduces Tree Exit Verification (TEV), an exact, level‑parallel speculative decoding method that uses a single exit‑node decision and a bonus‑token decision to verify draft continuations. By deriving a canonical exit law, the authors show that TEV achieves the same acceptance as optimal verifiers while reducing verifier‑stage latency by 15%. They also propose Exit‑Guided Draft‑Tree Training (ExitTrain), which improves draft trees and increases average output‑block length by 13%, leading to a 14% end‑to‑end speedup over DDTree across dialogue, code, and mathematical reasoning tasks.

By Shijing Hu, Xuancheng Ren, Zhihui Lu, Pan Zhou
arXiv AI
Oct 12

ReplaySSM: Cache SSM Inputs, Not State

ReplaySSM is a technique that redefines how state-space models (SSMs) handle their recurrent state by caching SSM inputs instead of the state itself. It unrolls the recurrence into a checkpoint plus a buffer of recent inputs, reducing state memory traffic by about half and enabling efficient speculative rollback. The method also allows separate precision levels for checkpoints, improving output accuracy while accelerating decoding in hybrid models such as Mamba-2 and Gated DeltaNet.

By Ze-Wei Liou, Harris Nover, Amey Naik, Andrii Skliar, Igor Shovkun, Ido Shahaf, Tri Dao
arXiv Computation and Language
Nov 6

TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding

TACG (Trajectory‑Aware Commit Gating) is a training‑free decoding strategy for diffusion language models that decouples token proposals from the decision to commit them. It evaluates temporal support by comparing current logits to an exponential moving average of past logits, enforces identity persistence through a History Gate, and limits the number of new positions per step. In LLaDA evaluations, TACG achieves 51.02 % mean accuracy with 67.82 mean model evaluations, outperforming native confidence‑based decoding and improving accuracy by 8.3 points at matched commit counts.

By Chengcheng Wang, Tingzhang Luo, Wenhao Li, Jianyuan Guo, Chang Xu
arXiv Machine Learning
Oct 3

LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models

LumoTree is a verifier designed for hybrid language models that ensures a single coherent continuation across recurrent, convolution, and attention states. It executes recurrent paths in parallel, reuses state tiles, and coordinates replay, history gathering, and cache remapping through a shared logical tree. Experiments demonstrate exact candidate‑selection parity and recurrent agreement within error bounds, and a Qwen3.8‑27B deployment achieved 25.63 pooled tokens/s on a single NVIDIA DGX Spark.

By Zhiyuan Ma
arXiv Machine Learning
Jul 20

DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts

arXiv:2605. 15422v3 Announce Type: replace Abstract: Modern RL post-training methods such as GRPO and DAPO train on N response sequences of R tokens sampled from a shared prompt of P tokens, but standard FlashAttention replicates all P prompt tokens N times across both forward and backward passes -- duplicating compute and memory on identical hidden states.

By Jiading Gai, Shuai Zhang, Xiang Song, Bernie Wang, George Karypis