The paper introduces a self‑distillation method that leverages answer‑only data to improve large language models (LLMs) after training. By training an inverse reasoning LLM to generate latent chain‑of‑thought traces conditioned on the answer, the authors can perform offline distillation that significantly boosts performance on hard prompts for tasks like Countdown and GSM8K. Additionally, the inverse model serves as an accurate verifier, enhancing best‑of‑n selection accuracy.
arXiv:2608. 15445v1 Announce Type: new Abstract: When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization.
arXiv:2606. 15576v1 Announce Type: cross Abstract: Reinforcement learning from verifiable rewards assigns a single scalar to each rollout, leaving token-level credit assignment underspecified in long reasoning traces.
The paper investigates why reinforcement learning with verifiable rewards (RLVR) reduces the diversity of solutions in reasoning tasks. By analyzing the Countdown task, the authors show that RLVR contracts the solution space mainly at the entrance—before the first arithmetic operation—causing a 67% drop in solution coverage. They demonstrate that providing an unselected entrance prefix or applying entrance‑targeted interventions can restore or even improve coverage without harming accuracy.
arXiv:2608. 06243v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level.
The paper introduces masked self‑distillation, a post‑training framework that trains a language model to internalize portions of its own intermediate reasoning traces. By varying the fraction of trace internalized, the authors demonstrate that models can achieve higher inference efficiency and improved task performance on math and graph‑coloring problems. Experiments on Qwen3‑4B and Qwen3‑8B show that the method generalizes well to in‑domain out‑of‑distribution cases without catastrophic forgetting, and that supervised fine‑tuning alone can reduce trace length at the expense of generalization.
arXiv:2605.11467v2 Announce Type: replace-cross Abstract: Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of *reasoning theater*: deliberativ...
arXiv:2605.10194v2 Announce Type: replace Abstract: On-policy self-distillation (OPSD) uses a model as its own teacher under privileged context, providing token-level supervision on the model's own r...
arXiv:2505. 24273v2 Announce Type: replace Abstract: Recent advancements in large language models (LLMs) suggest that reinforcement learning (RL) effectively internalizes search strategies, yielding significant improvements on challenging reasoning tasks through extended chains of thought.
Reinforcement learning from verifiable rewards assigns a single scalar to each rollout, leaving token-level credit assignment underspecified in long reasoning traces. On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information, producing a dense per-token signal.
arXiv:2608. 01522v1 Announce Type: new Abstract: Teaching a language model a skill it has not mastered is obstructed by three recurring difficulties: training data is scarce, ground-truth reasoning traces are usually unavailable, and models often exhibit an apparent ceiling beyond which additional data yields no further improvement.
arXiv:2609.37633v1 Announce Type: cross Abstract: The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards...
The paper introduces EvoResearcher, a training‑free, inference‑time protocol that enables a frozen large language model to perform cost‑bounded self‑reflection and early stopping. By iterating through generate → self‑critique → revise steps until a maximum depth or a CONFIRMED sentinel is reached, the model can self‑verify its answers within a strict compute budget. The protocol incorporates four self‑reflective meta‑reward components—correctness, efficiency, reflection depth, and tool‑call diversity—implemented as prompt‑level mechanisms, and is validated on Big‑Bench Hard, GSM8K, and MATH benchmarks, achieving comparable accuracy while terminating 82‑88% of items early with only about 2.1 generations per question.