The paper introduces CIG‑SFT, a supervised fine‑tuning method that uses conditional information gain to evaluate how much each demonstrated token reduces uncertainty about the task. By comparing token likelihoods from a frozen reference model with and without task input, the method derives a KL‑regularized projection of the current policy, yielding a scalar‑weighted SFT objective whose gradients match the projected target. Experiments on mathematical reasoning and code‑generation benchmarks show that CIG‑SFT outperforms standard SFT and adaptive fine‑tuning baselines, especially in generalizing to challenging evaluation distributions, without requiring extra data or online rollouts.
By Yuxuan Gu, Kequan Niu, Xiaocheng Feng, Yun Li, Yiqiao Yang, Bing Qin
arXiv:2605.30327v2 Announce Type: replace-cross
Abstract: Frontier reasoning models are produced by post-training base language models with reinforcement learning. Recent work has challenged this by...
By Felix Zhou, Anay Mehrotra, Quanquan C. Liu
arXiv:2609.37066v1 Announce Type: cross
Abstract: Post-training is central to mathematical reasoning in modern large language models (LLMs), but endpoint pass@1 alone underidentifies what has changed...
By Hongyang Li, Yiming Zhu, Xiao Li, Caesar Wu, Said Mammar, Pascal Bouvry
The paper introduces DIAL-OPD, a token‑selection strategy for on‑policy distillation that weights reward magnitude by the logarithmic mean of teacher and student probabilities, controlled by a parameter β. Experiments on four teacher‑student pairs across seven mathematical reasoning benchmarks show that retaining only 40 % of tokens with DIAL‑OPD yields up to 5.25 % higher mean accuracy than vanilla OPD and doubles AIME25 Pass@16, outperforming even stronger full‑token baselines. The study demonstrates that careful supervision allocation can surpass gains from simply scaling teacher size.
By Anhao Zhao, Haoran Xin, Junlong Tong, Yingqi Fan, Xuan Lu, Ping Nie, Wenjie Li, Xiaoyu Shen
The paper reports that in on‑policy distillation for large language models, reasoning performance can be improved by supervising only a tiny fraction of generated tokens—sometimes just one or two tokens per reasoning trajectory, about 0.05% of all tokens. This sparse supervision consistently matches or exceeds full‑token training across nine teacher‑student setups on mathematical reasoning, and is also validated on coding reasoning, Llama models, and PPO‑based reinforcement learning with verifiable reward. The findings suggest that effective post‑training does not require token‑intensive supervision and may align more closely with natural learning processes that focus on critical reasoning steps.
By Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu, Pan Xu, Karim Bouyarmane
arXiv:2607. 22602v1 Announce Type: new Abstract: Inference-time scaling has emerged as a powerful paradigm for improving large language model reasoning, often delivering larger gains on difficult reasoning tasks than parameter scaling alone.
By Tingxin Yang, Zefeng Wang, Mengyue Wang, Xingcheng Zhou, Yunpu Ma
arXiv:2606. 01682v1 Announce Type: cross Abstract: Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths.
By Atoosa Chegini, Soheil Feizi
arXiv:2605. 11936v2 Announce Type: replace Abstract: Recent soft prompt research has tried to improve reasoning by inserting trained vectors into LLM inputs, yet whether the gain comes from the learned content or from the act of injection itself has not been carefully separated.
By Heejun Kim, Seungpil Lee, Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Taesup Kim, Sundong Kim
arXiv:2606. 19771v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning; however, it faces a fundamental optimization instability: uniform token updates precipitate entropy collapse, leading to premature convergence to suboptimal strategies, whereas excessive Shannon Entropy maximization can cause entropy explosion, driving blind exploration toward incoherent reasoning chains.
By Xuanzhi Feng, Zhengyang Li, Zeyu Liu, Haoxi Li, Yuming Jiang, Bing Guo, Jingcai Guo, Jie Zhang, Song Guo
arXiv:2606. 31575v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a powerful tool for propelling Large Language Models (LLMs) beyond imitation-based training towards more robust reasoning capabilities.
By Outongyi Lv, Yanzhao Zheng, Yuanwei Zhang, Zhenghao Huang, Xingjun Wang, Baohua Dong, Hangcheng Zhu, Yingda Chen
arXiv:2605.28295v2 Announce Type: replace
Abstract: Reinforcement learning with verifiable rewards (RLVR) trains reasoning models without labeled trajectories, using groups of verifier-scored rollout...
By Soeun Kim, Albert No
Online Self-Weighted Fine‑Tuning (OSW‑FT) augments standard supervised fine‑tuning by adding online, trajectory‑level weighting: for each query the model estimates its current success rate from a small number of inference‑only rollouts and rescales the SFT loss accordingly. The method keeps the optimization direction anchored to the expert trajectory while adapting the update magnitude online, and it is shown to be unbiased for any finite rollout count with a convergence analysis. Across Qwen3 models from 0.6B to 4B, OSW‑FT consistently outperforms plain SFT on challenging benchmarks such as AIME, achieving a favorable compute‑performance trade‑off with only two online rollouts.
By Haiquan Wen, Yiwei He, Bei Peng, Guangliang Cheng