← Back to all news
arXiv AI By Yijing Ke, Tianle Cai, Ruosong Wang

Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning

Read the original on arXiv AI →

The paper investigates how aligning rounding locations between inference and training can stabilize reinforcement learning for large language models using BF16 precision. By retaining selected intermediates in FP32 until their BF16 outputs in three key regions—RoPE, SwiGLU, and residual RMSNorm—the authors prevent training collapse across four Qwen-family models. The approach keeps the inference engine, matrix‑multiplication and attention kernels, and learning objective unchanged.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
1d ago

bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix

The paper identifies a hidden mismatch in reinforcement‑learning fine‑tuning of large language models: when training uses fp32 master weights but rollouts are served from a bf16 copy produced by round‑to‑nearest conversion, most per‑step updates are too small to affect the served weights, causing a lag of 50–150 steps. The authors propose Fair Cast, a one‑line change that exports the master weights with an unbiased rounding rule (stochastic rounding or error feedback) and computes the loss on the served copy, eliminating the lag without adding hyperparameters and incurring only a modest computational cost. Across multiple models and optimizers, Fair Cast consistently outperforms round‑to‑nearest, improves checkpoint consistency, and reveals the implicit regularization effect of the lag.

By Andrew Gao, William Gao
llmsreinforcement-learning
More like this →
arXiv Machine Learning
Jul 20

QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides

arXiv:2607. 15810v1 Announce Type: new Abstract: Rollout generation is a major bottleneck in Reinforcement Learning (RL) for Mixture-of-Experts (MoE) Large Language Models, motivating low-precision rollout acceleration such as FP8.

By Zhengyang Zhuge, Hao Yu, Xin Wang, Zheng Li, Yizhong Cao, Dayiheng Liu, Jianwei Zhang
llmsreinforcement-learningefficiencybenchmarkssafety
More like this →
arXiv Machine Learning
Sep 10

One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control

The paper introduces OSOL, a method for mitigating higher‑order interference in multi‑domain reinforcement learning. OSOL selects a focus domain each iteration, uses token‑level footprints from the previous checkpoint to rank rebound risk, and applies an adaptively scaled correction to the GRPO update. Experiments on Qwen3‑30B‑A3B show a 5.7% improvement over the best baseline without higher‑order differentiation.

By Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Guojun Yin, Wei Lin, Ran He
llmsreinforcement-learningsafety
More like this →
arXiv Machine Learning
6d ago

InterQ-RL: Interaction-Aware Correction of Execution Mismatch in Quantized Agentic RL

InterQ‑RL is a new method for multi‑turn agent reinforcement learning that addresses execution mismatch caused by low‑precision quantized rollouts. It corrects the mismatch by applying bounded execution‑prefix adjustments over the policy history, allocating correction strength across action segments within a fixed displacement budget, and does so without extra model forward passes. Experiments on ALFWorld, WebShop, and ScienceWorld with 4B‑30B backbones show that InterQ‑RL achieves the best quantized performance, matching BF16 within 0.7 points and surpassing it in two settings while maintaining or improving learning speed.

By Yingbo Hao, Song Liu, Guanghua Yu, Zhaojing Zhou, Jianchen Zhu, Yi Zou
agentsreinforcement-learningefficiencysafety
More like this →