arXiv Machine Learning October 23, 2026 By Xuyang Zhong, Jiuding Yang, Yiwen Guo, Chen Liu

MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback

Read the original on arXiv Machine Learning →

MOER is a mix‑policy optimization framework that blends expert guidance with self‑exploration in reinforcement learning. It injects a decaying expert chain‑of‑thought prefix into prompts, progressively annealing the prefix length to encourage exploration, and uses a selective rollback mechanism to replace failed completions with the full expert trace, ensuring correctness. The method employs a harmonized mix‑policy objective that optimizes on‑policy tokens with a GRPO loss while supervising both expert prefix and completion tokens with a trust‑region SFT loss, leading to stable training and superior performance on mathematical and general reasoning tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language

Sep 14

Expert-Space Exploration in MoE Reinforcement Learning

arXiv:2609.13058v1 Announce Type: new Abstract: Reinforcement learning (RL) has become central to post-training of large language models. Recent advances in RL for Mixture-of-Experts (MoE) models hav...

By Hongyi He, Zhenghao Lin, Xiao Liu, Peng Cheng, Yan Lu, Yeyun Gong

llms reinforcement-learning

arXiv Machine Learning

Oct 7

Structuring MoE Expert Selection for Agentic Reinforcement Learning

arXiv:2610.07332v1 Announce Type: new Abstract: Long-horizon LLM agents are frequently implemented using sparse mixture-of-experts (MoE) models, yet the co-design of agentic behavior and MoE structur...

By Bolian Li, Ting-Yao Hu, Cheng-Yu Hsieh, Sanjoy Chowdhury, Oncel Tuzel, Raviteja Vemulapalli

llms agents reinforcement-learning benchmarks

arXiv AI

Sep 4

FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

arXiv:2609. 03241v1 Announce Type: cross Abstract: A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode.

By Zixun Huang, Kishan Panaganti, Haitao Mi, Leowei Liang

arXiv Machine Learning

Jul 9

UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

arXiv:2607. 06987v1 Announce Type: new Abstract: Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs).

By Chongyu Fan, Pengfei Liu, Jingjia Huang, Sijia Liu, Yi Lin

llms reinforcement-learning multimodal

arXiv AI

Oct 7

Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding

Rationale-Guided Policy Optimization (RGPO) is a reinforcement‑learning framework that adaptively uses ground‑truth rationale information to scaffold a language model’s reasoning process. Instead of treating reference solutions as fixed imitation targets, RGPO temporarily incorporates rationales to help the model generate better responses, then reverts to unguided learning with higher‑reward, model‑generated solutions. Experiments in both language‑only and vision‑language tasks show that RGPO consistently outperforms RLVR baselines, with ablation studies confirming that adaptive rationale guidance is a key factor in its success.

By Hoang Phan, Minh Pham, Chau Pham, Chinmay Hegde, Trung Le, Qi Lei

llms reinforcement-learning efficiency multimodal benchmarks

arXiv Machine Learning

Jun 25

FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning

arXiv:2605. 20256v2 Announce Type: replace Abstract: Reinforcement learning has become a cornerstone for aligning and unlocking the reasoning capabilities of large-scale models.

By Xikai Zhang, Yongzhi Li, Likang Xiao, Yingze Zhang, Yanhua Cheng, Quan Chen, Peng Jiang, Wenjun Wu, Liu Liu

reinforcement-learning

One email a morning, machine-written

Related stories

Expert-Space Exploration in MoE Reinforcement Learning

Structuring MoE Expert Selection for Agentic Reinforcement Learning

FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding

FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning