The paper investigates Evolution Strategies (ES) as a memory‑efficient post‑training method for large language model (LLM) reasoning. It demonstrates that ES outperforms Group Relative Policy Optimization (GRPO) by achieving broader reasoning coverage, improving Pass@K metrics, and avoiding entropy collapse. The study also reveals that ES’s performance gains stem from sparse, high‑magnitude parameter updates, do not cause catastrophic forgetting, and can be combined with GRPO in a sequential training strategy.
By Yunpeng Ba, Zhi Zheng, Yue Xie, Jiaqing Li, Xialiang Tong, Tao Zhong, Mingxuan Yuan, Zhichao Lu, Xuyang Wu, Zhenkun Wang
arXiv:2603. 17310v2 Announce Type: replace Abstract: Large Language Models (LLMs) with extended reasoning capabilities often generate verbose and redundant reasoning traces, incurring unnecessary computational cost.
By Chengwei Wei, Jung-jae Kim, Longyin Zhang, Shengkai Chen, Nancy F. Chen
arXiv:2608.28771v1 Announce Type: new
Abstract: Large reasoning models achieve strong performance on complex tasks by generating extended chain-of-thought (CoT) traces via reinforcement learning with...
By Xin Jiang, Minhao Wang, Wen Wu, Zhentao Xie, Shangheng Du, Jinxin Shi, Jiabao Zhao
arXiv:2606. 10184v1 Announce Type: cross Abstract: Group Relative Policy Optimization (GRPO) relies on the diversity of $K$ rollouts within each group; otherwise, the group-mean advantage $A^{(k)} = r^{(k)} - \mu_r$ collapses to zero.
By Wooil Jung
The paper introduces Echo-GRPO, a method that rewrites privileged reasoning traces into a model’s own idiolect to align off‑policy supervision with the student policy’s vocabulary. By preserving semantics through Dual‑Reference Decoding, Echo‑GRPO mitigates gradient clipping on critical reasoning tokens and improves reasoning distillation. The approach is instantiated as VideoEcho‑R1 for video reasoning, yielding consistent gains across multiple multimodal LLM backbones and benchmarks, and it can be applied as a plug‑in to both RL and supervised fine‑tuning frameworks.
By Ji Soo Lee, Jinyoung Park, Seohyun Lee, Jongha Kim, Joonmyung Choi, Jinsung Yoon, Hyunwoo J. Kim
arXiv:2608. 05541v1 Announce Type: new Abstract: Evolution Strategy (ES) is a promising alternative to gradient-based fine-tuning for resource-constrained Large Language Model (LLM) reasoning.
By Yu Gu, Zhi Zheng, Yunpeng Ba, Xialiang Tong, Mingxuan Yuan, Zhenkun Wang
arXiv:2605.30327v2 Announce Type: replace-cross
Abstract: Frontier reasoning models are produced by post-training base language models with reinforcement learning. Recent work has challenged this by...
By Felix Zhou, Anay Mehrotra, Quanquan C. Liu
The paper introduces a method to enhance large language model (LLM) exploration in Reinforcement Learning with Verifiable Rewards (RLVR) by guiding the target model with partial reasoning trajectories from smaller, weaker language models. This weak-model guidance disrupts over‑confidence, preserves generative diversity, and mitigates entropy collapse without extra fine‑tuning or complex reward designs. Experiments on mathematical benchmarks show consistent improvements over vanilla RLVR, especially as the number of allowed attempts ($k$) increases, indicating broader reasoning coverage.
By Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao
arXiv:2601. 09085v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has become a standard approach for training mathematical reasoning models; however, its reliance on multiple completions per prompt makes training computationally expensive.
By Kangda Wei, Ruihong Huang
arXiv:2607. 09693v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become the dominant paradigm for improving the reasoning capabilities of large language models, but it requires expensive training, curated data, and reward signals.
By Zibin Meng, Peng Xie, Kani Chen
The paper reports that in on‑policy distillation for large language models, reasoning performance can be improved by supervising only a tiny fraction of generated tokens—sometimes just one or two tokens per reasoning trajectory, about 0.05% of all tokens. This sparse supervision consistently matches or exceeds full‑token training across nine teacher‑student setups on mathematical reasoning, and is also validated on coding reasoning, Llama models, and PPO‑based reinforcement learning with verifiable reward. The findings suggest that effective post‑training does not require token‑intensive supervision and may align more closely with natural learning processes that focus on critical reasoning steps.
By Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu, Pan Xu, Karim Bouyarmane
arXiv:2606. 24994v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward.
By Wenyang Hu, Junxiang Jia, Zhen Shu, Daniel Dahlmeier, See-Kiong Ng, Bryan Kian Hsiang Low