arXiv:2608.24479v1 Announce Type: new Abstract: Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for...
The paper introduces Self‑Sparse Speculative Rollouts (S$^3$R), a method that uses sparse attention to propose tokens and then verifies them with full attention, thereby reducing KV‑cache traffic while preserving the dense‑policy distribution for reinforcement learning rollouts. S$^3$R eliminates the need for a separate drafter by sharing policy parameters between drafting and verification, and a KV‑aware scheduler limits cache thrashing to maintain parallelism. Experiments on long‑context mathematical reasoning with the Qwen3 model show that S$^3$R achieves 1.35–1.39× higher end‑to‑end training throughput with comparable task performance.
arXiv:2610.01133v1 Announce Type: cross Abstract: Scaling reasoning typically spends more compute on reinforcement learning (RL) or on inference. We show that a completed RL training history can yiel...
Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for data-limited replay. Through controlled experimen...
arXiv:2608. 07719v1 Announce Type: new Abstract: Offline reinforcement learning repeatedly trains policies from a fixed transition pool, making redundant data costly across seeds and hyperparameters, while naive subsampling can remove rare transitions needed for long-horizon credit assignment.
The paper proposes CANOPY, a minimalist reinforcement learning protocol that addresses two common pitfalls—signal starvation and policy drift—in outcome‑only RL for long‑horizon interactive tasks. By scaling same‑task exploration, keeping updates on‑policy, and anchoring updates with KL divergence, CANOPY enables a Qwen3‑14B agent to achieve top leaderboard results on the AppWorld coding benchmark without auxiliary supervision or elaborate scaffolding. The approach also improves performance on SWE‑bench for a Qwen3.5‑9B model.
arXiv:2606. 08854v1 Announce Type: cross Abstract: Standard Reinforcement Learning with Verifiable Rewards (RLVR) training allocates a fixed rollout budget to every query, without regard for what each query's difficulty means for the current policy.
SCARL (Sequence Confidence-Aligned Reinforcement Learning) addresses a misalignment between a reinforcement learning policy’s confidence and its reward signal in language-model reasoning. By reshaping each advantage with an alignment gap that reflects the difference between a response’s group-relative confidence and the outcome it certifies, SCARL drives responses with equal advantage toward equal confidence and mitigates entropy collapse. Experiments on Qwen3‑4B‑Base show a 2.4‑point lift in Pass@32 over DAPO and consistent improvements across multiple estimators and backbones while preserving policy entropy.
arXiv:2607. 27787v1 Announce Type: new Abstract: Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability.
arXiv:2606. 15333v1 Announce Type: cross Abstract: LLM unlearning has emerged as a cost-effective alternative to full retraining for removing hazardous knowledge from pretrained models while preserving general utility.
arXiv:2601. 22448v2 Announce Type: replace Abstract: RLVR has become a standard recipe for training LLMs on reasoning tasks with verifiable outcomes, but when rollout generation dominates the cost, efficiency hinges on which prompts are sampled and when.
arXiv:2605. 05481v2 Announce Type: replace Abstract: We revisit a classic "chicken-and-egg" problem in reinforcement learning: to safely improve a policy, the value function must be accurate on the state-visitation distribution of the updated policy.