arXiv Machine Learning October 22, 2026 By Chengxi Li, Hanlin Wu, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Jingjing Liu, Hao Zhou

EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping

Read the original on arXiv Machine Learning →

EPIC introduces a new PPO-style clipping method that operates at the prefix level rather than token level, using the exact reverse-KL coordinate to regulate policy drift during autoregressive generation. By reformulating generation as a finite-horizon Markov decision process, the authors derive a performance-difference identity that identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each policy-improvement step. Experiments on off‑policy reinforcement learning with a sparse mixture‑of‑experts language model demonstrate that EPIC consistently outperforms standard and sequence‑level clipping baselines on mathematical‑reasoning benchmarks while remaining competitive with routing‑replay alternatives.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 15

Rethinking the Trust Region in LLM Reinforcement Learning

arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.

By Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, Wee Sun Lee

llms reinforcement-learning fine-tuning efficiency

arXiv AI

Jul 1

Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index

arXiv:2606. 31575v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a powerful tool for propelling Large Language Models (LLMs) beyond imitation-based training towards more robust reasoning capabilities.

By Outongyi Lv, Yanzhao Zheng, Yuanwei Zhang, Zhenghao Huang, Xingjun Wang, Baohua Dong, Hangcheng Zhu, Yingda Chen

llms reinforcement-learning benchmarks

arXiv Computation and Language

Sep 22

What are Key Factors for Updates in RL for LLM Reasoning?

arXiv:2606.22570v2 Announce Type: replace Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a promising framework for enhancing the reasoning ability of large language mo...

By Peidong Wang, Demi Wang, Xufang Luo, Jiahang Xu, Xiaocui Yang, Shi Feng, Yuqing Yang, Dongsheng Li

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 9

Rethinking the Divergence Regularization in LLM RL

arXiv:2606. 09821v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a key component of post-training large language models (LLMs).

By Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee, Liefeng Bo, Tianyu Pang

llms reinforcement-learning

arXiv AI

Jun 10

Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning

arXiv:2606. 10968v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has become standard for improving LLM reasoning.

By Renjie Mao, Xiangxin Zhou, Lvfang Tao, Yixin Ding, Yu Shi, Yongguang Lin, Yuheng Wu, Honglin Zhu, Qian Qiu, Wenxi Zhu

llms reinforcement-learning

arXiv AI

Jun 30

PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF

arXiv:2606. 29758v1 Announce Type: cross Abstract: Reinforcement Learning from Human Feedback (RLHF) for Large Language Models increasingly relies on critic-free methods as a practical alternative to actor--critic training.

By Doo Hwan Hwang, Kee-Eung Kim

llms reinforcement-learning benchmarks

One email a morning, machine-written

Related stories

Rethinking the Trust Region in LLM Reinforcement Learning

Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index

What are Key Factors for Updates in RL for LLM Reasoning?

Rethinking the Divergence Regularization in LLM RL

Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning

PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF