arXiv AI
Splicing Hints into Reasoning Traces: Learning from Demonstrator Failures
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 9
Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards
arXiv:2605. 03862v4 Announce Type: replace Abstract: Reinforcement learning with verifiable rewards has become a common way to improve explicit reasoning in large language models, but final-answer correctness alone does not reveal whether the reasoning trace is faithful, reliable, or useful to the model that consumes it.
arXiv Machine Learning
3d ago
TraceForge: Distilling with Your Own Thinking
TraceForge introduces a two‑stage training method that generates reasoning traces in the style of a target model family, using supervised learning followed by adversarial refinement. The system produces traces that are judged as authentic with scores comparable to real traces, and when applied to teacher answers, it improves performance on coding benchmarks such as HumanEval+ and MBPP+. The approach addresses the limitation of existing models that only return answers without detailed reasoning, enabling more transparent and learnable outputs.
arXiv AI
Jul 17
Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models
arXiv:2607. 14552v1 Announce Type: cross Abstract: A standard recipe for distilling the reasoning ability of large language models (LLMs) is to sample chains of thought from the model, keep those that reach the correct final answer, and fine-tune on the survivors.
arXiv Machine Learning
Oct 15
Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data
The paper introduces a self‑distillation method that leverages answer‑only data to improve large language models (LLMs) after training. By training an inverse reasoning LLM to generate latent chain‑of‑thought traces conditioned on the answer, the authors can perform offline distillation that significantly boosts performance on hard prompts for tasks like Countdown and GSM8K. Additionally, the inverse model serves as an accurate verifier, enhancing best‑of‑n selection accuracy.
arXiv Machine Learning
Sep 3
Cliff: Learning Process Rewards from the First Mistake
Cliff is a reward‑shaping method for reinforcement learning with verifiable rewards that identifies the first mistake in a language model’s reasoning process using an off‑the‑shelf teacher. By splitting each rollout into a correct prefix and an incorrect suffix, Cliff assigns token‑level advantages—positive for correct tokens and negative for the rest—providing fine‑grained supervision. Across 12 scenarios, Cliff outperforms on‑policy distillation by 15% and standard GRPO by 7%, even when the teacher is only modestly capable.
arXiv AI
Oct 12
A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
The paper examines on‑policy self‑distillation, where a teacher scores a student’s rollout to provide dense token‑level credit. Experiments show that the raw scoring signals neither localize errors nor effectively penalize incorrect tokens, because locating errors is a search problem that the teacher is not explicitly asked to perform. Introducing a critique‑conditioned approach—first generating a critique of the rollout and then conditioning the teacher on it—produces signals that accurately localize errors, correct erroneous spans, and guide the policy toward backtracking tokens, improving performance on math problems and in cold‑start GRPO settings.