arXiv:2606. 09887v1 Announce Type: cross Abstract: Reinforcement learning (RL) for large language models usually supervises reasoning with scalar outcome rewards, such as binary correctness.
arXiv:2607. 28582v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort.
Cliff is a reward‑shaping method for reinforcement learning with verifiable rewards that identifies the first mistake in a language model’s reasoning process using an off‑the‑shelf teacher. By splitting each rollout into a correct prefix and an incorrect suffix, Cliff assigns token‑level advantages—positive for correct tokens and negative for the rest—providing fine‑grained supervision. Across 12 scenarios, Cliff outperforms on‑policy distillation by 15% and standard GRPO by 7%, even when the teacher is only modestly capable.
Cliff is a reward‑shaping method for reinforcement learning with verifiable rewards that identifies the first mistake in a language model’s reasoning process using an off‑the‑shelf teacher. It splits each rollout into a correct prefix and an incorrect suffix, assigning positive token‑level advantages to the prefix and negative feedback to the suffix. Across 12 scenarios, Cliff improves reasoning performance, outperforming on‑policy distillation by 15% and standard GRPO by 7%, even with modest teachers.
SIPO (Self‑Instructing Policy Optimization) unifies reinforcement learning with on‑policy self‑distillation by using a contrastive self‑teacher to generate token‑level credit signals. The method samples multiple rollouts per prompt, pairs each with a reference answer and its mistakes, and uses the difference in teacher log‑probabilities to provide dense feedback while still respecting the overall task reward. Experiments on reasoning and code‑generation benchmarks show that SIPO outperforms both RLVR and OPSD baselines without requiring an external teacher or extra generation steps.
Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning.
arXiv:2603.11321v3 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Rewards improves reasoning in large language models, yet on-policy learning often suffers from cold-st...
arXiv:2607. 04412v1 Announce Type: new Abstract: Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals.
arXiv:2606. 17024v1 Announce Type: new Abstract: Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model.
arXiv:2606. 19327v1 Announce Type: new Abstract: Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards.
arXiv:2607. 18955v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation.
OPDSearch+ introduces a two‑stage distillation framework for search‑augmented reasoning that eliminates the need for task‑specific teacher fine‑tuning. In the first stage, a frozen off‑the‑shelf instruct model guides a student through live search interactions using a per‑position forward KL objective, transferring reasoning decomposition and evidence integration skills. The second stage refines this student with reinforcement learning, achieving performance surpassing RL alone and outperforming all prior 3B‑parameter baselines on seven QA benchmarks, including 13.1% improvement on HotpotQA and 8.5% on 2WikiMultihopQA.
arXiv:2608. 12764v1 Announce Type: cross Abstract: Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory, which is far too sparse for effective credit assignment.