arXiv:2606. 20008v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards has become a central tool for improving the reasoning ability of large language models, but current methods face a trade-off between simplicity and credit assignment.
arXiv:2606. 18810v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps.
arXiv:2607. 25659v1 Announce Type: new Abstract: Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria.
arXiv:2606. 25451v1 Announce Type: new Abstract: Estimating token-level advantages in reinforcement learning (RL) for language models remains challenging because scaling up episodic experience collection is expensive.
arXiv:2608.21501v1 Announce Type: new Abstract: Credit assignment in large-language-model reinforcement learning (LLM RL) can be separated into three objects: evidence about success, a transport oper...
arXiv:2608. 02181v1 Announce Type: new Abstract: Proximal Policy Optimization (PPO) for large language models typically trains its critic by mean-squared-error (MSE) regression on scalar value targets.
arXiv:2609.36178v1 Announce Type: cross Abstract: Group Relative Policy Optimization (GRPO) has become a promising approach for training large language model agents. However, its uniform assignment o...
The paper introduces Actor‑Critic with Action Chunking (AC2), a method that assigns credit to short action chunks instead of entire trajectories, enabling policy updates without waiting for terminal rewards. AC2 employs local readiness, reference solutions, and 10k‑token chunks to make critic‑based credit assignment reliable. Experiments on Qwen3‑4B with FineProofs‑RL show AC2 surpasses GRPO’s peak validation score while using 2.5× fewer decoding FLOPs and fewer training steps.
The paper introduces TASPO, a method that transforms privileged information (PI) into outcome‑grounded action credit for language‑model agents. TASPO constructs decision‑applicable PI from verified successful experience, aggregates PI‑induced likelihood shifts at the executable‑action level, and converts relative action support into positive, bounded, mean‑preserving weights on the original trajectory advantage. Experiments on three agentic benchmarks show TASPO improves over GRPO by 10.6% and generalizes better to unseen tasks, while reducing supervision mismatch and stabilizing policy optimization.
arXiv:2609.40360v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions...
arXiv:2609.36802v1 Announce Type: new Abstract: A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning...
The paper introduces Best‑Practice Critic Optimization (BPCO), a stable and efficient recipe for training a critic in reinforcement learning for large language models. BPCO combines DPPO, bounded value predictions, Monte Carlo targets, unnormalized policy advantages, and length‑adaptive advantage estimation, allowing the critic to be conditioned on hidden reward information. Experiments on mathematical reasoning tasks with models from 1.5B to 30B parameters show that BPCO consistently outperforms a strong critic‑based baseline and matches or exceeds group‑based methods while sampling only one response per prompt.