arXiv Machine Learning

PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs

arXiv AI
Aug 11

Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training

arXiv:2608. 09217v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become a central post-training paradigm for eliciting reasoning capabilities in large language models, yet uniform task sampling allocates compute without regard to differences in how tasks respond to optimization.

By Ting Zhou, Zhenqing Ling, Daoyuan Chen, Qianli Shen, Yilun Huang, Ying Shen, Yaliang Li
Hugging Face Trending Papers
Aug 10

Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training

Reinforcement learning (RL) has become a central post-training paradigm for eliciting reasoning capabilities in large language models, yet uniform task sampling allocates compute without regard to differences in how tasks respond to optimization. Existing task-valuation methods mostly rely on snapshot-based signals such as current pass rate or reward, which estimate how solvable a task is under the current policy.

arXiv Machine Learning
Jun 25

Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents

arXiv:2606. 26080v1 Announce Type: new Abstract: Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale.

By Changdae Oh, Wendi Li, Seongheon Park, Samuel Yeh, Tanwi Mallick, Sharon Li
arXiv AI
Sep 10

ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search

ReST‑RL introduces a unified Reinforced Self‑Training (ReST) policy‑value framework that enhances large language model (LLM) reasoning by combining an optimized ReST‑style GRPO algorithm with a value‑guided search (VM‑MCTS). The ReST‑GRPO component reshapes trajectory distributions to increase reward variance and expose policies to more informative partial states, improving training efficiency. VM‑MCTS trains a Value Model from self‑collected Monte‑Carlo Tree Search targets and uses it during inference to provide precise process signals and verification scores, boosting reasoning accuracy across coding benchmarks and out‑of‑domain math and science tasks.

By Sining Zhoubian, Dan Zhang, Jie Tang
arXiv AI
1d ago

Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO

The paper introduces FastRL, a reinforcement learning framework designed to enhance the efficiency of Group Relative Policy Optimization (GRPO) and its variants. FastRL employs an advantage-aware pruning strategy that retains high-advantage trajectories while preserving gradient diversity, and an adaptive rollout sampling mechanism that adjusts sampling scale during training based on historical pruning data. Experiments show that FastRL can be integrated into GRPO, DAPO, and GSPO, yielding a 2.07× speedup on Geometry3K and GeoQA8K-R1V and a 1.64% accuracy improvement on visual reasoning benchmarks.

By Jiahua Yang, Zhiwei Yang, Xianpeng Zhang, Dongyu Chen, Xing Chen, Tianhuang Su, Haonan Lu, Quanlong Guan, Kai Tang, Chuangchuang Wang