The paper investigates Evolution Strategies (ES) as a memory‑efficient post‑training method for large language model (LLM) reasoning. It demonstrates that ES outperforms Group Relative Policy Optimization (GRPO) by achieving broader reasoning coverage, improving Pass@K metrics, and avoiding entropy collapse. The study also reveals that ES’s performance gains stem from sparse, high‑magnitude parameter updates, do not cause catastrophic forgetting, and can be combined with GRPO in a sequential training strategy.
By Yunpeng Ba, Zhi Zheng, Yue Xie, Jiaqing Li, Xialiang Tong, Tao Zhong, Mingxuan Yuan, Zhichao Lu, Xuyang Wu, Zhenkun Wang
arXiv:2609.38104v1 Announce Type: new
Abstract: Power-sharpened sampling is an inference-time alternative to reinforcement-learning (RL) post-training for enhancing reasoning in large language models...
By Panagiotis Theodoropoulos, Nan Jiang, Xintong Duan, Ali Hasan, Yuriy Nevmyvaka, Evangelos A. Theodorou, Wei Deng
The paper reports that in on‑policy distillation for large language models, reasoning performance can be improved by supervising only a tiny fraction of generated tokens—sometimes just one or two tokens per reasoning trajectory, about 0.05% of all tokens. This sparse supervision consistently matches or exceeds full‑token training across nine teacher‑student setups on mathematical reasoning, and is also validated on coding reasoning, Llama models, and PPO‑based reinforcement learning with verifiable reward. The findings suggest that effective post‑training does not require token‑intensive supervision and may align more closely with natural learning processes that focus on critical reasoning steps.
By Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu, Pan Xu, Karim Bouyarmane
arXiv:2604. 08477v2 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved reasoning in formal domains such as mathematics and code, but extending these gains beyond STEM remains challenging.
By Ashima Suvarna, Kendrick Phan, Mehrab Beikzadeh, Hritik Bansal, Saadia Gabriel
arXiv:2609.37066v1 Announce Type: cross
Abstract: Post-training is central to mathematical reasoning in modern large language models (LLMs), but endpoint pass@1 alone underidentifies what has changed...
By Hongyang Li, Yiming Zhu, Xiao Li, Caesar Wu, Said Mammar, Pascal Bouvry
arXiv:2607. 09693v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become the dominant paradigm for improving the reasoning capabilities of large language models, but it requires expensive training, curated data, and reward signals.
By Zibin Meng, Peng Xie, Kani Chen
The paper introduces a method to enhance large language model (LLM) exploration in Reinforcement Learning with Verifiable Rewards (RLVR) by guiding the target model with partial reasoning trajectories from smaller, weaker language models. This weak-model guidance disrupts over‑confidence, preserves generative diversity, and mitigates entropy collapse without extra fine‑tuning or complex reward designs. Experiments on mathematical benchmarks show consistent improvements over vanilla RLVR, especially as the number of allowed attempts ($k$) increases, indicating broader reasoning coverage.
By Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao
The paper investigates how different forms of compressed chain‑of‑thought (CoT) reasoning—Explicit, Composed, and Implicit—affect large language model (LLM) performance after supervised fine‑tuning (SFT). Using a synthetic compositional reasoning task, the authors show that coarser CoT requires more SFT data, that Composed and Implicit CoT benefit more from data scaling (with Composed also benefiting from repetition), and that reinforcement learning with verifiable rewards (RLVR) can decompose compressed steps learned during SFT. Additionally, unidirectional CoT ordering improves generalization on longer sequential tasks.
By Kohsei Matsutani, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
arXiv:2605.30327v2 Announce Type: replace-cross
Abstract: Frontier reasoning models are produced by post-training base language models with reinforcement learning. Recent work has challenged this by...
By Felix Zhou, Anay Mehrotra, Quanquan C. Liu
The paper introduces Trimmed Logit-Gap SFT (TrimSFT), a token-level reweighting strategy that adjusts supervised fine-tuning loss based on the logit gap between the correct token and its strongest competitor. TrimSFT trims supervision from tokens that are either already mastered (large logit gap) or poorly supported (small or negative logit gap), focusing learning on tokens with intermediate logit gaps. Experiments on six base models across five mathematical reasoning benchmarks show that TrimSFT consistently outperforms standard SFT, achieving the best average performance on five of six models and up to +26.9 points on MATH500.
By Yaning Jia, Chunhui Zhang, Wenxuan Xu, Xingjian Diao, Xiaoyuan Wang, Soroush Vosoughi
arXiv:2606. 17024v1 Announce Type: new Abstract: Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model.
By Violet Xiang, Amrith Setlur, Chase Blagden, Nick Haber, Aviral Kumar
arXiv:2607. 26119v1 Announce Type: new Abstract: Large reasoning models trained via reinforcement learning (RL) have been increasingly shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks; Yet the mechanistic basis for this advantage remains unclear.
By Antyabha Rahman, Akshaj Gurugubelli, Omar Ankit, Kevin Zhu, Aishwarya Balwani