arXiv AI By Shuibai Zhang, Junhyuck Kim, Gyeongman Kim, Jaewoong Cho

Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?

Read the original on arXiv AI →

The paper investigates whether the importance ratio is essential for stable reinforcement learning (RL) in large language models (LLMs). It introduces Likelihood‑Gated Policy Optimization (LGPO), which removes the need to compute old‑policy likelihoods by using a likelihood‑based gating mechanism to enforce a soft trust region. Experiments show that eliminating the importance sampling correction does not hurt stability, while removing the trust region can cause collapse; LGPO matches or exceeds GRPO’s performance, reduces training time by about 18%, and remains stable in fully on‑policy settings where GRPO fails.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Nov 5

Stabilizing Off-policy LLM Optimization with Prefix Importance Ratio

The paper addresses instability in off‑policy reinforcement learning for large language models (LLMs) caused by large policy lag. It shows that the correct correction for the prefix‑marginal policy gradient is the cumulative prefix importance ratio, which token‑level weighting fails to capture. To improve stability, the authors propose Minimum Prefix Ratio (MinPRO), a surrogate objective that uses the minimum token‑level ratio in the preceding prefix, and demonstrate its effectiveness on dense and mixture‑of‑experts LLMs across mathematical reasoning benchmarks.

By Shiye Lei, Zhihao Cheng, Dacheng Tao
arXiv Machine Learning
Sep 18

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

The paper introduces QUATRO, a reinforcement‑learning approach for fine‑tuning large language models that enforces trust‑region constraints directly rather than relying on heuristic clipping. By deriving a principled objective, QUATRO provides explicit control over policy updates and stabilizes entropy during training. Experiments on mathematical reasoning benchmarks demonstrate that QUATRO maintains stable training even with higher learning rates and increased policy staleness.

By Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Sookyung Kim, Joonseok Lee, Jaemoo Choi