arXiv AI October 1, 2026 By Tianle Zhong, Neiwen Ling, Yifan Pi, Zijun Wei, Tianshu Yu, Geoffrey Fox, Peng Wu, Xiao Yu

Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference

The paper investigates Training‑Inference Mismatch (TIM) in large‑language‑model reinforcement learning, where rollout generation and policy optimization produce differing token probabilities despite identical model weights. By creating a zero‑mismatch diagnostic setting called VeXact, the authors isolate TIM and demonstrate that even minor token‑level numerical disagreements can trigger training collapse. They further show that TIM alters the effective optimization problem and propose remedies to mitigate its impact.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jun 30

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

arXiv:2606. 29526v1 Announce Type: new Abstract: Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.

By Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng

llms reinforcement-learning

arXiv AI

Jul 14

ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

arXiv:2607. 10481v1 Announce Type: cross Abstract: Reinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs), yet the training process remains notoriously fragile.

By Kexin Huang, Junkang Wu, Jinda Lu, Shuo Yang, Chiyu Ma, Jiancan Wu, Xiang Wang, Xiangnan He, Guoyin Wang, Jingren Zhou

llms reinforcement-learning benchmarks

arXiv AI

Sep 15

Diagnosing Faults in Reinforcement Learning Simulators and World Models with Canonical Polynomial Invariants

arXiv:2609.13194v1 Announce Type: cross Abstract: A large literature builds physical structure into learned dynamics on the premise that models respecting the underlying physics predict better. We te...

By Tesfay Zemuy Gebrekidan, Hadush Hailu Gebrerufael

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 16

Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

arXiv:2606. 15333v1 Announce Type: cross Abstract: LLM unlearning has emerged as a cost-effective alternative to full retraining for removing hazardous knowledge from pretrained models while preserving general utility.

By Zirui Pang, Chenlong Zhang, Haosheng Tan, Zhuoran Jin, Jiaheng Wei, Zixin Zhong

llms reinforcement-learning benchmarks

arXiv Machine Learning

Sep 10

One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control

The paper introduces OSOL, a method for mitigating higher‑order interference in multi‑domain reinforcement learning. OSOL selects a focus domain each iteration, uses token‑level footprints from the previous checkpoint to rank rebound risk, and applies an adaptively scaled correction to the GRPO update. Experiments on Qwen3‑30B‑A3B show a 5.7% improvement over the best baseline without higher‑order differentiation.

By Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Guojun Yin, Wei Lin, Ran He

llms reinforcement-learning safety

arXiv Machine Learning

Jun 25

Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It

arXiv:2606. 26027v1 Announce Type: cross Abstract: Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for enhancing model capabilities.

By Yupu Hao, Zhuoran Jin, Huanxuan Liao, Kang Liu, Jun Zhao

llms agents reinforcement-learning fine-tuning

One email a morning, machine-written

Related stories

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

Diagnosing Faults in Reinforcement Learning Simulators and World Models with Canonical Polynomial Invariants

Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control

Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It