arXiv Computation and Language

What are Key Factors for Updates in RL for LLM Reasoning?

arXiv AI
Sep 15

HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments

HISPO introduces a segment‑level policy‑optimization technique for reinforcement learning with verifiable rewards, creating entropy‑derived contiguous segments during rollout and applying clipped importance‑sampling correction at this granularity. It bridges the gap between token‑level and sequence‑level corrections, offering a middle‑ground approach for credit assignment in long‑form mathematical reasoning. Evaluated on Qwen3‑1.7B‑Base across six benchmarks, HISPO consistently outperforms or matches the strongest baselines in Pass@8 and Acc@8 metrics, notably improving AIME25 scores over GRPO and GSPO.

By Quoc-Vinh Lai-Dang, Hyo-Sang Shin
arXiv AI
Oct 28

When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning

The paper introduces Policy-Compatible Reinforcement Learning (PICO), a method that addresses the learning cliff in reinforcement learning with verifiable rewards (RLVR) by performing distribution-constrained joint decoding. PICO guides token-level decoding with expert data while preserving the model’s token probabilities, and uses a dynamic top‑k mechanism to keep sampling within a finite divergence budget. The resulting trajectories are verified and policy‑compatible, and are jointly optimized with on‑policy rollouts using a mixed‑behavior objective that mitigates distribution mismatch via importance sampling, leading to consistent performance gains over standard RLVR and prior off‑policy baselines on diverse reasoning datasets and agentic benchmarks.

By Zehao Wang, Hanzhu Chen, Runquan Gui, Zhe Wang, Lin Yang
arXiv Computation and Language
Aug 28

Boosting LLM Exploration via Weak-Model Guidance in RLVR

The paper introduces a method to enhance large language model (LLM) exploration in Reinforcement Learning with Verifiable Rewards (RLVR) by guiding the target model with partial reasoning trajectories from smaller, weaker language models. This weak-model guidance disrupts over‑confidence, preserves generative diversity, and mitigates entropy collapse without extra fine‑tuning or complex reward designs. Experiments on mathematical benchmarks show consistent improvements over vanilla RLVR, especially as the number of allowed attempts ($k$) increases, indicating broader reasoning coverage.

By Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao
Hugging Face Trending Papers
Oct 27

When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning

The paper introduces Policy-Compatible Reinforcement Learning (PICO), a method that combines distribution-constrained joint decoding with on-policy rollouts to address the learning cliff in reinforcement learning with verifiable rewards (RLVR). PICO guides token-level decoding using expert data while preserving the model’s relative token probabilities, and employs a dynamic top‑k mechanism to keep sampling within a finite divergence budget. This approach yields verified, policy‑compatible trajectories that are jointly optimized with on‑policy rollouts, leading to consistent performance gains over standard RLVR and prior off‑policy baselines across diverse reasoning datasets and agentic benchmarks.

arXiv AI
Nov 5

Stabilizing Off-policy LLM Optimization with Prefix Importance Ratio

The paper addresses instability in off‑policy reinforcement learning for large language models (LLMs) caused by large policy lag. It shows that the correct correction for the prefix‑marginal policy gradient is the cumulative prefix importance ratio, which token‑level weighting fails to capture. To improve stability, the authors propose Minimum Prefix Ratio (MinPRO), a surrogate objective that uses the minimum token‑level ratio in the preceding prefix, and demonstrate its effectiveness on dense and mixture‑of‑experts LLMs across mathematical reasoning benchmarks.

By Shiye Lei, Zhihao Cheng, Dacheng Tao