arXiv AI September 11, 2026 By Xingwei Gan, Ying Zhu

Complementing reinforcement learning with SFT through logit averaging in the post training of LLMs

The paper proposes a new technique that averages the logits of a frozen reference policy (such as one obtained via supervised fine‑tuning) and a trainable policy, integrating this approach into Group Relative Policy Optimization (GRPO). Unlike methods that use Kullback–Leibler regularization or a critic, the proposed method couples the policies solely through logit averaging, aiming to combine the reasoning strengths of the trainable policy with the formatting benefits of the reference policy. Experiments on MATH, cn‑k12, and MMLU demonstrate that this approach achieves higher or comparable accuracy to the standard KL‑regularized GRPO.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jun 2

Stabilizing Policy Optimization via Logits Convexity

arXiv:2603. 00963v2 Announce Type: replace Abstract: While reinforcement learning (RL) has been central to the recent success of large language models (LLMs), RL optimization is notoriously unstable, especially when compared to supervised fine-tuning (SFT).

By Hongzhan Chen, Tao Yang, Yuhua Zhu, Shiping Gao, Xiaojun Quan, Ting Yao

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 19

VIMPO: Value-Implicit Policy Optimization for LLMs

arXiv:2606. 20008v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards has become a central tool for improving the reasoning ability of large language models, but current methods face a trade-off between simplicity and credit assignment.

By Zhewei Kang, Aosong Feng, Sergey Levine, Dawn Song, Xuandong Zhao

llms reinforcement-learning benchmarks

arXiv AI

Aug 25

How to Train a Critic Stably and Efficiently

The paper introduces Best‑Practice Critic Optimization (BPCO), a stable and efficient recipe for training a critic in reinforcement learning for large language models. BPCO combines DPPO, bounded value predictions, Monte Carlo targets, unnormalized policy advantages, and length‑adaptive advantage estimation, allowing the critic to be conditioned on hidden reward information. Experiments on mathematical reasoning tasks with models from 1.5B to 30B parameters show that BPCO consistently outperforms a strong critic‑based baseline and matches or exceeds group‑based methods while sampling only one response per prompt.

By Penghui Qi, Xiangxin Zhou, Wee Sun Lee

llms reinforcement-learning

arXiv AI

Aug 26

Best Practice Critic Optimization

arXiv:2608.23566v2 Announce Type: replace-cross Abstract: Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for...

By Penghui Qi, Xiangxin Zhou, Wee Sun Lee

llms reinforcement-learning

arXiv AI

Jul 22

RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts

arXiv:2607. 18470v1 Announce Type: cross Abstract: Group Relative Policy Optimization (GRPO) has shown strong effectiveness in reinforcement learning from verifiable feedback, where sampled rollouts can be compared within a group using task-provided correctness signals.

By Yuxin Xiong, Xunyi Jiang, Rohan Surana, Xintong Li, Sheldon Yu, Nikki Lijing Kuang, Ryan A. Rossi, Jingbo Shang, Tong Yu, Julian McAuley, Junda Wu

reinforcement-learning fine-tuning safety

arXiv Machine Learning

Aug 18

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

arXiv:2608. 16739v1 Announce Type: new Abstract: Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy.

By Siddarth Venkatraman, Matthieu Dinot, Laurence Aitchison

llms reinforcement-learning efficiency

One email a morning, machine-written

Related stories

Stabilizing Policy Optimization via Logits Convexity

VIMPO: Value-Implicit Policy Optimization for LLMs

How to Train a Critic Stably and Efficiently

Best Practice Critic Optimization

RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts

Le Critique: Privileged Value Functions for LLM Reinforcement Learning