The paper investigates end‑to‑end FP4 reinforcement learning for Mixture‑of‑Experts large language models and finds that existing recovery techniques for dense models fail to improve accuracy in the MoE setting. The authors attribute this to the discrete routing decisions in MoE, which make training‑inference policy alignment highly sensitive to asymmetry in recovery methods. They introduce MoEResQ, a symmetric quantization‑recovery recipe that restores alignment and significantly narrows the accuracy gap to full precision.
arXiv:2610.07767v1 Announce Type: new Abstract: Reinforcement learning (RL) for post-training large language models (LLMs) incurs substantial computation and memory overhead during rollout generation...
arXiv:2607. 15810v1 Announce Type: new Abstract: Rollout generation is a major bottleneck in Reinforcement Learning (RL) for Mixture-of-Experts (MoE) Large Language Models, motivating low-precision rollout acceleration such as FP8.
arXiv:2609.13058v1 Announce Type: new Abstract: Reinforcement learning (RL) has become central to post-training of large language models. Recent advances in RL for Mixture-of-Experts (MoE) models hav...
arXiv:2606. 00395v1 Announce Type: cross Abstract: Mixture of Experts (MoE) Large Language Models (LLMs) achieve strong performance at scale.
arXiv:2610.07332v1 Announce Type: new Abstract: Long-horizon LLM agents are frequently implemented using sparse mixture-of-experts (MoE) models, yet the co-design of agentic behavior and MoE structur...
arXiv:2606. 29526v1 Announce Type: new Abstract: Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.
The paper introduces Router Drift Control (RDC), a method for mitigating off‑policy degradation in Mixture‑of‑Experts reinforcement learning. RDC uses the disagreement between rollout and training routers to detect stale routing decisions and selectively rejects high‑drift responses, thereby avoiding the gradient bias introduced by Rollout Routing Replay (R3). Experiments show that RDC improves validation accuracy, especially in asynchronous training settings, outperforming both R3 and Truncated Importance Sampling (TIS).
The paper introduces Ban&Pick, a post‑training, plug‑and‑play routing strategy for Sparse Mixture‑of‑Experts large language models. It identifies and reinforces a small group of highly influential experts while dynamically pruning redundant ones, leading to accuracy gains across math, code, and reasoning benchmarks. Experiments on DeepSeek and Qwen3 show notable performance improvements and a 1.25× inference speedup without retraining or architectural changes.
arXiv:2606. 05688v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale foundation models efficiently by activating only a subset of experts for each token, but their large number of expert parameters still makes quantization essential for practical deployment.
The paper introduces Calibrated Clipping, a dynamic method to align FP8 quantization bounds with high‑precision BF16 distributions, thereby mitigating training instability in full‑pipeline FP8 reinforcement learning for large language models. It identifies that compounded FP8 noise distorts importance ratios, causing entropy surges and garbled outputs. Experiments across GRPO and DAPO algorithms on 8B‑32B models show the technique restores performance to BF16 levels.
InterQ‑RL is a new method for multi‑turn agent reinforcement learning that addresses execution mismatch caused by low‑precision quantized rollouts. It corrects the mismatch by applying bounded execution‑prefix adjustments over the policy history, allocating correction strength across action segments within a fixed displacement budget, and does so without extra model forward passes. Experiments on ALFWorld, WebShop, and ScienceWorld with 4B‑30B backbones show that InterQ‑RL achieves the best quantized performance, matching BF16 within 0.7 points and surpassing it in two settings while maintaining or improving learning speed.
The paper introduces OnQ, a low‑bit recovery pipeline for quantized large language models that uses on‑policy distillation and candidate‑wise Dynamic KL to address trajectory divergence and distributional misalignment. OnQ starts with block‑wise calibration, then lets the quantized student generate responses and receive full‑precision teacher supervision on its own prefixes, dynamically weighting forward and reverse KL losses per candidate. Experiments on Qwen3 models show that OnQ outperforms baselines on nine benchmarks, especially long‑horizon reasoning, and produces shorter, less truncated responses.