The paper introduces Self‑Sparse Speculative Rollouts (S$^3$R), a method that uses sparse attention to propose tokens and then verifies them with full attention, thereby reducing KV‑cache traffic while preserving the dense‑policy distribution for reinforcement learning rollouts. S$^3$R eliminates the need for a separate drafter by sharing policy parameters between drafting and verification, and a KV‑aware scheduler limits cache thrashing to maintain parallelism. Experiments on long‑context mathematical reasoning with the Qwen3 model show that S$^3$R achieves 1.35–1.39× higher end‑to‑end training throughput with comparable task performance.
arXiv:2606. 18967v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a representative post-training paradigm for LLMs, enabling strong reasoning and agentic capabilities.
arXiv:2606. 08446v1 Announce Type: cross Abstract: Despite being powerful, reinforcement learning with verifiable rewards (RLVR) induces extremely long COT, making it computationally expensive.
The paper introduces APEX, a learned controller that improves speculative decoding for large language models by dynamically selecting the best speculation strategy and adjusting draft depth during generation. APEX uses request‑level routing among EAGLE‑3, n‑gram, and draft‑model speculation, and block‑level depth adaptation based on causal decoding signals and verifier feedback. Integrated into vLLM and tested with Qwen3‑8B, APEX achieves up to 5.24× speedup over autoregressive decoding while reducing wasted tokens by 41% compared to fixed‑depth n‑gram speculation.
arXiv:2606. 00144v1 Announce Type: cross Abstract: Speculative decoding speeds up autoregressive decoding by using a drafter to propose multiple tokens that a verifier validates in parallel.
arXiv:2608. 04962v1 Announce Type: new Abstract: Reinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but autoregressive rollout generation remains a major efficiency bottleneck.
The paper proposes CANOPY, a minimalist reinforcement learning protocol that addresses two common pitfalls—signal starvation and policy drift—in outcome‑only RL for long‑horizon interactive tasks. By scaling same‑task exploration, keeping updates on‑policy, and anchoring updates with KL divergence, CANOPY enables a Qwen3‑14B agent to achieve top leaderboard results on the AppWorld coding benchmark without auxiliary supervision or elaborate scaffolding. The approach also improves performance on SWE‑bench for a Qwen3.5‑9B model.
The paper addresses the memory bottleneck in reinforcement learning for large language models caused by the large Key-Value (KV) cache during rollout phases. It highlights that while KV cache compression can reduce memory usage, it introduces a significant off‑policy bias that standard statistical corrections cannot adequately mitigate. The authors argue that even tiny compression errors are amplified by RL’s instability, leading to inefficient learning.
arXiv:2609.14717v1 Announce Type: cross Abstract: Speculative decoding accelerates LLM inference by verifying multiple drafted tokens in parallel, allowing a single target forward pass to accept seve...
The paper introduces the Expected Decoding Rounds (EDR) objective, a theoretical framework that models speculative decoding as a Markov reward process and directly optimizes the number of decoding rounds. EDR weights local rejection costs by state occupancies, eliminating auxiliary hyperparameters and enabling unbiased stochastic optimization via temporal‑difference gradients. Applying EDR to fine‑tune state‑of‑the‑art draft models (DSpark and DFly) improves mean accepted length and outperforms prior training objectives across nine benchmarks in math reasoning, code generation, and chat.
Reinforcement learning (RL) has become a key component in modern large language models, yet the rollout stage remains the key bottleneck in RL training pipelines. Although Multi-Token Prediction (MTP) offers a natural solution to accelerate rollouts through speculative decoding, many studies have observed that MTP acceptance rates degrade significantly during RL training, leading to limited speedup performance.
arXiv:2607. 21535v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel.
arXiv:2607. 26253v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal.