arXiv:2512. 23075v5 Announce Type: replace-cross Abstract: Policy gradient methods for Large Language Models optimize a policy $\pi_\theta$ via a surrogate objective computed from samples of a rollout policy $\pi_{\text{roll}}$.
By Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Qian Liu, Baoxiang Wang
The paper introduces a fidelity‑aware post‑training framework for coding and terminal agents that preserves token and control fidelity by sampling from original prompts, eliminating spurious model calls, and limiting loss computation to verifiable token spans. It also proposes Certified Divergence Proximal Policy Optimization (C‑DPPO), which provides tight two‑sided total variation certification bounds, adaptive‑K rules, budget‑aware sequence guarantees, and error‑robust policy masking. Experiments on Baize5B and Baize10B models show a consistent +3.0‑point performance improvement over standard DPPO, with certificate audits confirming full operational coverage.
By Cheng Li, Jiexiong Liu, Yixuan Chen, Chi Hong
The paper presents a convergence framework for deep $V$‑learning over a finite horizon $H$, deriving explicit bounds on policy loss by decomposing the Bellman update error into six residuals. It shows how $L^s$ concentrability controls expected $L^1$ loss, quantifies the impact of shared sampling across horizon levels, and provides optimal and near‑optimal sample allocations for statistical error rates. The work also establishes sharp action‑gap bounds under a margin condition, transfers optimal‑gap results to frozen‑iterate gaps, and offers consistency guarantees for generative‑reset approximate‑ERM procedures with exact action scores.
By Yury Kolomeytsev
arXiv:2610.00888v1 Announce Type: cross
Abstract: Multi-token prediction (MTP) improves the throughput of autoregressive generation by enabling the language model to draft multiple next tokens per fo...
By Prachi Badarayani, Aidan Jay, Chenghui Zhou, Dayquan Julienne, Yuan Gao, Tianwei Chen, George Zerveas, Ishmam Zabir, Xiren Zhou, Chris Quirk, Xia Song
The paper investigates how different estimators of the reverse Kullback–Leibler (KL) divergence used as a regularization term in reinforcement learning (RL) training of large language models (LLMs) affect training stability and downstream performance. By analyzing gradient bias across various estimator configurations, the authors demonstrate that biased gradients can cause training instabilities, while unbiased configurations improve performance on both in‑domain and out‑of‑domain tasks. Experiments on Qwen2.5‑7B, Llama‑3.1‑8B‑Instruct, and Qwen3‑4B‑Instruct‑2507 confirm these findings and show that KL regularization also stabilizes off‑policy RL training in asynchronous setups.
By Vedant Shah, Johan Obando-Ceron, Vineet Jain, Brian Bartoldson, Bhavya Kailkhura, Sarthak Mittal, Glen Berseth, Pablo Samuel Castro, Yoshua Bengio, Esmeralda S. Whitammer, Moksh Jain, Siddarth Venkatraman, Aaron Courville
arXiv:2607. 25271v1 Announce Type: cross Abstract: Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data.
By Tian Qin, Kimia Hamidieh, David Alvarez-Melis