The paper introduces CONCUR, an advantage estimator designed for multi‑task, multi‑reward reinforcement learning with large language models. CONCUR replaces standardized scores of gated auxiliary rewards with adjusted scores that ensure higher primary rewards never receive lower scores and that responses at the primary reward ceiling receive equal scores. Two variants, CONCUR‑T and CONCUR‑A, are evaluated on models across math, code, instruction following, and writing tasks, outperforming all baselines and the approach of dropping all gated auxiliaries.
arXiv:2610.20497v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual...
SIPO (Self‑Instructing Policy Optimization) unifies reinforcement learning with on‑policy self‑distillation by using a contrastive self‑teacher to generate token‑level credit signals. The method samples multiple rollouts per prompt, pairs each with a reference answer and its mistakes, and uses the difference in teacher log‑probabilities to provide dense feedback while still respecting the overall task reward. Experiments on reasoning and code‑generation benchmarks show that SIPO outperforms both RLVR and OPSD baselines without requiring an external teacher or extra generation steps.
arXiv:2610.34862v1 Announce Type: cross Abstract: LLM post-training often optimizes multiple objectives with unequal priorities, such as correctness and reasoning efficiency. However, the widely used...
arXiv:2609.26355v1 Announce Type: new Abstract: Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted ma...
The paper investigates how combining multiple rewards during large‑language‑model post‑training can lead to a failure called multi‑reward hacking, where the overall reward improves while the primary task performance worsens. The authors show that symmetric reward aggregation can cause the policy to favor auxiliary rewards that are more reliably obtained, regardless of initialization. They propose Trust‑Anchored Gating (TAG), which selects an anchor reward channel and gates auxiliary credit based on anchor satisfaction, thereby preventing auxiliary‑only outcomes and restoring primary‑task performance. Experiments on Qwen and Llama models demonstrate that TAG significantly mitigates degradation seen with additive, constrained, or normalized aggregation, achieving higher pass rates and shorter responses compared to baseline methods.
arXiv:2609.00213v1 Announce Type: new Abstract: Reinforcement learning fine-tuning of large language models increasingly adopts multiple reward dimensions, including verifiable rules, task-specific e...
The paper introduces Density-Aware Reward Aggregation (DARA), a method that adjusts reward weighting in multi-reward reinforcement learning based on the density of active rewards within rollout batches. By applying an inverse-square-root density correction, DARA gives more weight to less frequently active rewards, enabling faster learning of targeted behaviors. Experiments on tool calling and mathematical reasoning demonstrate that DARA achieves comparable final performance while reducing training steps by up to 26% and 65% respectively.
The paper introduces Best‑Practice Critic Optimization (BPCO), a stable and efficient recipe for training a critic in reinforcement learning for large language models. BPCO combines DPPO, bounded value predictions, Monte Carlo targets, unnormalized policy advantages, and length‑adaptive advantage estimation, allowing the critic to be conditioned on hidden reward information. Experiments on mathematical reasoning tasks with models from 1.5B to 30B parameters show that BPCO consistently outperforms a strong critic‑based baseline and matches or exceeds group‑based methods while sampling only one response per prompt.
The paper introduces Residual Advantage (RA), a new guidance method for reinforcement learning with verifiable rewards (RLVR) that treats the teacher–student probability residual as a bounded one‑step reward, subtracts the student’s state value to form an advantage, and centers it within each response. RA preserves the verifier’s mean label while redistributing credit among response steps, and its extension CoRA updates the teacher model using verifier advantages to adapt guidance to the student’s attempts. Experiments with Qwen3 models show that RA combined with GRPO or REINFORCE++ consistently outperforms teacher‑only on‑policy distillation across three mathematical benchmarks, improving macro Avg@8 and Pass@8 scores, with CoRA providing additional gains.
arXiv:2602. 18037v2 Announce Type: replace-cross Abstract: Reinforcement Learning from Human Feedback (RLHF) or Verifiable Rewards (RLVR) are two key steps in the post-training of modern Language Models (LMs).
arXiv:2608. 03223v1 Announce Type: cross Abstract: Agentic reinforcement learning enables LLM agents to learn through interaction, but sparse trajectory-level rewards reveal success without identifying which intermediate decisions deserve credit.
The paper introduces MRTPO, a method for assigning reward advantages at the token level in multi-reward reinforcement learning for language models. MRTPO uses the Jensen‑Shannon distance between next‑token distributions with and without a reward’s verifier to identify which tokens are relevant to each reward, thereby reallocating advantage without auxiliary models or annotations. Experiments on text‑to‑SQL, multi‑constraint instruction following, and tool calling show that MRTPO converges faster and outperforms existing methods.