Post-training large language models (LLMs) without real-world interaction feedback or human-labeled supervision remains challenging, particularly in specialized domains where expert annotations are costly to obtain. Recent annotation-free self-evolution methods address this by using the model's own outputs as supervision signals, constructing a teacher via additional context and aggregating predictions across multiple rollouts through majority voting to produce pseudo-labels.
RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) is a new method that builds a synthetic teacher from a language model’s own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor in parameter or logit space, RISE transforms sparse outcome-based updates into dense token-level targets without external models or privileged conditioning. The approach recursively refines the student model, combining RLVR and on‑policy distillation, and demonstrates superior performance across mathematical reasoning, STEM, code generation, and multi‑turn agentic tasks.
By Yang Li, Semih Yavuz, Shafiq Joty
SIPO (Self‑Instructing Policy Optimization) unifies reinforcement learning with on‑policy self‑distillation by using a contrastive self‑teacher to generate token‑level credit signals. The method samples multiple rollouts per prompt, pairs each with a reference answer and its mistakes, and uses the difference in teacher log‑probabilities to provide dense feedback while still respecting the overall task reward. Experiments on reasoning and code‑generation benchmarks show that SIPO outperforms both RLVR and OPSD baselines without requiring an external teacher or extra generation steps.
By Zhenrui Yue, Huimin Zeng, Yueqi Wang, Yaokun Liu, Fengran Mo, Jinghan Zhang, Mung Yao Jia, Gyuseok Lee, Yang Zhang, Na Wei, Dong Wang
The paper introduces SOLID, a framework that enables operations research language models to self-improve without relying on verified answers or external evaluators. SOLID uses solver-generated artifacts from the model’s own rollouts to create pseudo-references, clustering objectives and applying group-relative advantages for dense self-supervision. Experiments on multiple OR benchmarks show that SOLID enhances solution accuracy for both general-purpose and OR-tuned models compared to outcome-only training.
By Rui Zhu, Minglong Cao, Chenyu Zhou, Jianghao Lin, Dongdong Ge
arXiv:2607. 18955v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation.
By Qiye Cai, Yichuan Ma, Linyang Li, Peiji Li, Yongkang Chen, Qipeng Guo, Yicheng Zou, Tao Gui, Xiaocheng Feng, Bing Qin
arXiv:2606. 00172v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR), especially Group Relative Policy Optimization (GRPO), has been widely used to improve reasoning in large language models.
By Yang Li, Gongle Xue, Yijia Guo, Yuheng Yuan, Liwen Hu, Lei Ma
arXiv:2608. 03223v1 Announce Type: cross Abstract: Agentic reinforcement learning enables LLM agents to learn through interaction, but sparse trajectory-level rewards reveal success without identifying which intermediate decisions deserve credit.
By Ranxu Zhang, Guinan Chen, Chenshaodong, Jinghao Lin, Xiaozhou Xu, Sunzhe, Yanyong Zhang, Chao Wang
arXiv:2607. 23125v1 Announce Type: new Abstract: Post-training enables vision-language models (VLMs) to understand human instructions and perform various downstream tasks.
By Shuai Wang, Daoan Zhang, Zhe Tang, Hao Cheng, Jiaheng Wei
arXiv:2607. 05541v1 Announce Type: cross Abstract: Reinforcement Learning is commonly used to train large language models using environmental feedback.
By Muhammad Zain Amin, Kibele Sebnem Yildirim
arXiv:2608. 06243v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level.
By ZhiYan Hou, Xinyu Tang, Hongyan An, Jianjin Zhang, Weizhen Wang, Yunyun Han, Gengsheng Li, Xiangzhao Hao, Haiyun Guo, Wenbin Hu, Jinqiao Wang, Yafeng Deng
Co‑RL is a multi‑agent reinforcement learning framework that trains several decoupled models without shared parameters, using rewards generated by their peers. By increasing cohort diversity—through heterogeneous model families, varying sizes, and rephrased training samples—Co‑RL reduces self‑reinforcing feedback loops, preserves behavioral diversity, and prevents training collapse. Across both text‑only and multimodal benchmarks, Co‑RL outperforms base models and prior label‑free methods, achieving gains of 3.0‑8.6% on seven text benchmarks and 2.3‑7.2% on four multimodal benchmarks, while matching or surpassing supervised approaches without any ground‑truth labels.
By Yunhao Yang, Yuexin Bian, Yunjie Tian, Di Fu, Tianjin Huang, Yuanyuan Shi, Ziang Xiao, Nuno Vasconcelos, Yijiang Li
arXiv:2608.23493v1 Announce Type: new
Abstract: Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However,...
By Jialong Liu, Yuling Shi, Ning Yang, Xiaodong Gu, Zuchao Li