arXiv AI

Nice Fold or Hero Call: Learning Budget-Efficient Thinking under Policy-Dependent Solvability

The paper introduces Budget‑Efficient Thinking (BET), a two‑stage framework that treats adaptive reasoning as a computational investment, aligning solve‑or‑fold decisions with expected return rather than perceived difficulty. BET learns three distinct behaviors: concise short solves for easy queries, early abstention (nice fold) when further reasoning is unlikely to pay off, and allocating sufficient compute (hero call) for hard‑but‑solvable questions. Experiments on seven benchmarks with three base models show BET cuts reasoning tokens by 54% while boosting accuracy by up to 3.2%, and it transfers effectively to scientific QA and logical reasoning tasks.

arXiv AI
Aug 21

Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation

arXiv:2608. 20256v1 Announce Type: new Abstract: Reasoning language models trained with reinforcement learning typically operate under a fixed token budget rather than an explicitly adaptive one, which can lead to over-computation on easy problems and insufficient computation on difficult ones.

By Gijs Kassenaar, Zhao Yang, Vincent Fran\c{c}ois-Lavet
arXiv AI
Sep 18

When2Think: Learning Difficulty-Aware Length Control for Efficient Hybrid Reasoning Models

When2Think introduces a post‑training framework that dynamically allocates reasoning depth in Large Reasoning Models based on instance difficulty. The method uses Instance‑level Difficulty‑Aware Control (IDAC) to shape rewards with pre‑computed accuracy and token usage statistics, enabling stable, critic‑free optimization without learned reward models. Experiments on mathematical benchmarks show that When2Think improves accuracy‑efficiency trade‑offs, achieving higher Pass@3 scores while reducing token usage compared to baseline models.

By Jaejun Shim, HyunJin Kim, Young Jin Kim, JinYeong Bak
arXiv AI
Oct 21

PORA: Learning When More Reasoning Pays Off for LLM Agents

The paper introduces PORA, a method that teaches large language model agents to decide when to allocate more reasoning steps versus fewer, based on the expected downstream benefit. PORA trains by running each state twice—once with short reasoning and once with long reasoning—and learns to predict the value of extended reasoning. During inference, a controller uses this prediction and remaining budget information to choose the appropriate reasoning length without running both options, improving performance across diverse tasks.

By Bo Yan, Qi Zhang, Zhen Tan, Tianlong Chen, Song Wang
arXiv Machine Learning
Nov 2

Optimizing Anytime Reasoning via Budget Relative Policy Optimization

The paper introduces AnytimeReasoner, a framework that optimizes large language models for anytime reasoning by truncating the reasoning process to fit sampled token budgets and requiring the model to summarize the answer for verification. This approach adds verifiable dense rewards, improving credit assignment in reinforcement learning. The authors also propose Budget Relative Policy Optimization (BRPO), a variance reduction technique that enhances learning robustness and efficiency, and demonstrate that their method outperforms GRPO on mathematical reasoning tasks across various budgets and prior distributions.

By Penghui Qi, Zichen Liu, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin
arXiv Machine Learning
Nov 5

Reasoning as Compression: Unifying Budget Forcing via the Conditional Information Bottleneck

The paper proposes a new framework for efficient chain‑of‑thought (CoT) prompting called the Conditional Information Bottleneck (CIB). By treating reasoning as a lossy compression problem, CIB models the reasoning trace as a computational bridge that contains only the information about the response not directly available from the prompt, thereby overcoming a theoretical gap in naive Information Bottleneck approaches. The resulting reinforcement‑learning objective maximizes task reward while compressing completions under a semantic prior based on token surprisal, leading to reduced redundancy, preserved fluency and logic, and improved accuracy across models and tasks.

By Fabio Valerio Massoli, Andrey Kuzmin, Arash Behboodi
Hugging Face Trending Papers
Aug 5

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost. KV-cache compression is a common solution, yet existing reasoning-oriented methods apply a uniform policy across the trajectory and judge compression only by what it removes from the cache.

arXiv Machine Learning
Sep 21

Rewarding Efficient Reasoning Improves Abstention on Underspecified Tasks in Reasoning Models

The paper introduces a new reward, GRPO, that encourages large reasoning models (LRMs) to efficiently determine whether a task is solvable before generating a full chain of thought. Fine‑tuning 4B LRMs with this reward improves their ability to abstain from answering unanswerable prompts by an average of 12.8% while producing 44% shorter chains of thought. The approach also preserves the models’ overall answering performance.

By Polina Tsvilodub, Max H\"oth, Michael Franke, Bj\"orn Deiseroth, Carina Kauf