Selective Residual Adaptation (SRA) is a plug‑in framework that improves multi‑step reasoning in distilled large language models by correcting sparse distribution divergences between teacher and student along reasoning paths. It uses a Discrepancy Gate to spot divergence points and a Candidate Residual Adapter to calibrate top‑K logits, thereby avoiding changes to the student backbone or runtime teacher computation. Experiments on Qwen3 models from 0.6B to 4B show that SRA raises average Pass@16 by up to 4.37 percentage points while maintaining general capabilities.
arXiv:2603.00578v2 Announce Type: replace-cross Abstract: Long chain-of-thought~(CoT) reasoning improves the problem-solving capability of large reasoning models~(LRMs), but often incurs substantial...
BoundaryEvo introduces a two‑stage data‑model co‑evolution framework for distilling long‑chain reasoning from large teacher models into small language models. First, Capability Priming distills a small set of teacher trajectories to give the student an initial reasoning baseline. Then, Capability‑Boundary Co‑Evolution iteratively selects questions the student has not mastered but the teacher can solve, updating the training data as the student’s capability grows. With only 1,000 examples, BoundaryEvo improves Qwen3.5‑4B accuracy from 62.9 % to 75.3 % on IMO‑AnswerBench and from 26.8 % to 46.0 % on AMO‑Bench, outperforming static and dynamic baselines across 11 benchmarks.
SOD: Step-wise On-policy Distillation for Small Language Model Agents proposes a new framework that adaptively reweights distillation strength at each reasoning step based on step-level divergence. This approach mitigates cascading errors in tool-integrated reasoning by attenuating misleading teacher signals in high-divergence regions while preserving dense guidance where student and teacher align. Experiments on math, science, and code benchmarks show up to 20.86% improvement over the second-best baseline, with a 0.6B student scoring 26.13% on AIME 2025.
arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.
The paper introduces ACTD, an Anchor-Based Cross-Tokenizer Distillation method that aligns vocabularies and sequences to transfer reasoning capabilities from large language models to smaller students. It uses a novel anchor loss with residual regularization to reduce alignment noise and extends the approach to multiple teachers. Experiments on five reasoning benchmarks with three teachers show state‑of‑the‑art results, with the multi‑teacher variant outperforming existing baselines.
The paper introduces Universal Textual Teaching (UTT), a parameter‑update‑free framework that distills knowledge gaps between a Teacher and Student large language model into a reusable natural‑language artifact called a Primer. UTT identifies representative gap cases, then iteratively refines the Primer through a cycle of student attempts, prompter‑generated teaching instructions, teacher demonstrations, and synthesizer consolidation. Experiments on math (Omni‑MATH‑2) and code generation (KernelBench) show significant accuracy gains, outperforming prompt engineering and parameter‑based knowledge distillation, and the Primer generalizes across different Teacher‑Student pairs.
Artifact Distillation proposes a training‑free method for transferring knowledge from large frontier language models to smaller ones by having the teacher solve tasks and store its problem‑solving steps in an artifact bank that the student can query during inference. In experiments on LiveBench, a bank created by Claude Opus 4.5 enabled Claude Haiku 4.5 to surpass Claude Sonnet 4.5 on spatial reasoning and nearly match it on table joining, all at significantly lower cost. The same artifact bank also boosted Qwen3.5‑27B’s pooled score from 0.730 to 0.800, demonstrating that the approach works across different model families.
arXiv:2607. 14895v1 Announce Type: new Abstract: Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding.
arXiv:2606. 31048v1 Announce Type: cross Abstract: This paper investigates knowledge distillation from a large reasoning model (DeepSeek-R1) to a compact student model (Qwen2.
arXiv:2509. 21013v4 Announce Type: replace-cross Abstract: Given the prohibitive cost of pre-training large language models, it is essential to leverage smaller proxy models to optimize datasets before scaling up.
arXiv:2608.29623v1 Announce Type: cross Abstract: Recent advances in large reasoning models (LRMs) have shown strong performance on complex problems through long chain-of-thought (Long CoT) reasoning...
The paper introduces OnQ, a low‑bit recovery pipeline for quantized large language models that uses on‑policy distillation and candidate‑wise Dynamic KL to address trajectory divergence and distributional misalignment. OnQ starts with block‑wise calibration, then lets the quantized student generate responses and receive full‑precision teacher supervision on its own prefixes, dynamically weighting forward and reverse KL losses per candidate. Experiments on Qwen3 models show that OnQ outperforms baselines on nine benchmarks, especially long‑horizon reasoning, and produces shorter, less truncated responses.