arXiv Machine Learning

Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners

The paper introduces Thinking-Free Policy Initialization (TFPI), a simple adaptation to Reinforcement Learning with Verifiable Reward (RLVR) that removes explicit thinking content during training and inference. TFPI bridges long Chain-of-Thought distillation with standard RLVR, reducing token usage and improving performance. Experiments show that TFPI accelerates RL convergence, raises performance ceilings, and yields token‑efficient reasoning models, enabling a 4B model to achieve 89.0% accuracy on AIME24 and 65.5% on LiveCodeBench with under 4K H20 hours.

arXiv AI

5d ago

Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking

The paper introduces Root-token Policy Optimization (RPO), a framework that lets large reasoning models decide when to engage in explicit reasoning by training only the initial root token. RPO uses group-relative rewards and advantages to focus on this branching point, dramatically cutting training overhead and VRAM usage. Experiments across various model families show that RPO achieves difficulty-aware adaptive thinking with roughly 2% of the compute required by previous adaptive-reasoning methods.

By Taehyeon Kim, Hyunsoo Lee, Youngsoo Jang, Moontae Lee

arXiv AI

Sep 30

MetaCtrl: Your Large Language Models Can Reason Better and More Concisely with a Metacognitive Controller

arXiv:2609.37304v1 Announce Type: new Abstract: Large reasoning models improve performance on challenging problems by allocating additional computation before answering, but longer reasoning does not...

By Zhibin Wen, Tao Han, Lei Bai, Can Li, Yang Xu

llms reinforcement-learning benchmarks

arXiv Computation and Language

5d ago

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs

arXiv:2603.00578v2 Announce Type: replace-cross Abstract: Long chain-of-thought~(CoT) reasoning improves the problem-solving capability of large reasoning models~(LRMs), but often incurs substantial...

By Jie Cao, Tianwei Lin, Juekai Lin, Zhenxuan Fan, Yunqi Cao, Bo Yuan, Ziyuan Zhao, Rolan Yan, Wenqiao Zhang, Siliang Tang

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Sep 7

Extremely Sparse Supervision Incentivizes Reasoning Ability

The paper reports that in on‑policy distillation for large language models, reasoning performance can be improved by supervising only a tiny fraction of generated tokens—sometimes just one or two tokens per reasoning trajectory, about 0.05% of all tokens. This sparse supervision consistently matches or exceeds full‑token training across nine teacher‑student setups on mathematical reasoning, and is also validated on coding reasoning, Llama models, and PPO‑based reinforcement learning with verifiable reward. The findings suggest that effective post‑training does not require token‑intensive supervision and may align more closely with natural learning processes that focus on critical reasoning steps.

By Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu, Pan Xu, Karim Bouyarmane

llms reinforcement-learning efficiency

arXiv AI

Sep 30

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

arXiv:2609.37119v1 Announce Type: cross Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...

By Hongyang Li, Xiao Li, Caesar Wu, Said Mammar, Gr\'egoire Danoy, Pascal Bouvry

llms reinforcement-learning safety

arXiv Machine Learning

Jul 27

Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning

arXiv:2511. 02130v2 Announce Type: replace-cross Abstract: We propose Re-FORC, an adaptive reward prediction method that, given a query, enables prediction of the expected future rewards as a function of the number of future thinking tokens.

By Renos Zabounidis, Aditya Golatkar, Michael Kleinman, Alessandro Achille, Wei Xia, Stefano Soatto

arXiv Machine Learning

Sep 10

Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient Reasoning

arXiv:2604.02007v3 Announce Type: replace Abstract: Building general-purpose reasoning models using reinforcement learning with verifiable rewards (RLVR) across diverse domains has been widely adopte...

By Rafael Pardinas, Ehsan Kamalloo, David Vazquez, Alexandre Drouin

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 3

ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning

arXiv:2606. 03503v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) have achieved remarkable progress thanks to Reinforcement Learning with Verifiable Rewards (RLVR) on Chain-of-Thoughts (CoTs).

By Ziyan Liu, Xueda Shen, Yuzhe Gu, Songyang Gao, Kuikun Liu, Guangran Cheng, Chengqi Lyu, Dahua Lin, Wenwei Zhang, Kai Chen

reinforcement-learning benchmarks

arXiv AI

Jun 4

Mid-Think: Training-Free Intermediate-Budget Reasoning via Token-Level Triggers

arXiv:2601. 07036v2 Announce Type: replace-cross Abstract: Hybrid reasoning language models are commonly controlled through high-level Think/No-think instructions to regulate reasoning behavior, yet we found that such mode switching is largely driven by a small set of trigger tokens rather than the instructions themselves.

By Wang Yang, Debargha Ganguly, Xinpeng Li, Chaoda Song, Shouren Wang, Vikash Singh, Vipin Chaudhary, Xiaotian Han

llms reinforcement-learning fine-tuning

arXiv AI

Jun 9

Thinking-Based Non-Thinking: Solving the Reward Hacking Problem in Training Hybrid Reasoning Models via Reinforcement Learning

arXiv:2601. 04805v2 Announce Type: replace Abstract: Large reasoning models (LRMs) have attracted much attention due to their exceptional performance.

By Siyuan Gan, Jiaheng Liu, Boyan Wang, Tianpei Yang, Runqing Miao, Yuyao Zhang, Fanyu Meng, Junlan Feng, Linjian Meng, Jing Huo, Yang Gao

reinforcement-learning fine-tuning benchmarks

arXiv AI

Aug 3

ThinkReset: Learnable Intermediate Interface Construction for Bounded-Context Long-Horizon Reasoning

arXiv:2607. 28642v1 Announce Type: new Abstract: Long chain-of-thought reasoning improves performance on complex problems, but it also introduces redundancy accumulation, context overflow, and error anchoring.

By Fei Ding, Yongkang Zhang, Runhao Liu, Yuhao Liao, Zijian Zeng

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 9

From Shortcuts to Reasoning: Robust Post-Training of Theory of Mind with Reinforcement Learning

arXiv:2606. 09092v1 Announce Type: new Abstract: Theory of Mind (ToM) is a must-acquire skill for modern foundation model systems to operate effectively and safely in the real world.

By Jike Zhong, Yuxiang Lai, Ming Li, Yuheng Li, Wuao Liu, Behzad Dariush, Konstantinos Psounis, Shao-Yuan Lo

reinforcement-learning fine-tuning multimodal

Related stories