← Back to all news
arXiv Machine Learning By Yu Li, Yufa Zhou, Tian Lan

On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning

Read the original on arXiv Machine Learning →

The paper investigates the training‑inference mismatch that arises when large language models are trained with on‑policy reinforcement learning but inference is performed by a separate engine. It identifies two sources of mismatch—numerical discrepancies in the engine and policy lag due to stale weights—and shows that correcting one source can worsen the other. The authors propose a discrepancy‑lag correction (DLC) that centers numerical errors, clips lag ratios, and resynchronizes the sampler, achieving stable training on Qwen models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 8

COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning

The paper introduces COPC, a Coupled Off-Policy Correction method for asynchronous reinforcement learning of large language models. COPC coordinates policy-side and advantage-side corrections by combining token-level ratio masking with two-sided clipped-ratio weighting of TD residuals, addressing both policy mismatch and advantage staleness. Experiments show COPC outperforms existing asynchronous baselines on tool-integrated mathematical reasoning and search tasks, while maintaining training stability and minimal overhead.

By Zicheng Hu, Zhijian Zhou, Xuan Zhang, Yuchen Liu, Cheng Chen, Yuan Li, Qi Gu, Yan Feng, Hongyan Hao, Chao Qu
llmsreinforcement-learningsafety
More like this →
arXiv Machine Learning
4d ago

bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix

The paper identifies a hidden mismatch in reinforcement‑learning fine‑tuning of large language models: when training uses fp32 master weights but rollouts are served from a bf16 copy produced by round‑to‑nearest conversion, most per‑step updates are too small to affect the served weights, causing a lag of 50–150 steps. The authors propose Fair Cast, a one‑line change that exports the master weights with an unbiased rounding rule (stochastic rounding or error feedback) and computes the loss on the served copy, eliminating the lag without adding hyperparameters and incurring only a modest computational cost. Across multiple models and optimizers, Fair Cast consistently outperforms round‑to‑nearest, improves checkpoint consistency, and reveals the implicit regularization effect of the lag.

By Andrew Gao, William Gao
llmsreinforcement-learning
More like this →
arXiv Machine Learning
Sep 22

A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillation

The paper investigates selective on‑policy distillation, where a student model is trained only on token positions chosen by a selector. It demonstrates that the commonly used shared learning rate is not neutral: performance varies significantly with the learning rate for different selectors, leading to inconsistent comparisons. The authors attribute this selector‑rate entanglement to the selection process itself and recommend reporting the full arm‑by‑rate matrix for fair evaluation.

By Chencheng Zhu
fine-tuningefficiencybenchmarks
More like this →