Difficulty-Estimated Policy Optimization (DEPO) is a new framework that improves the efficiency and robustness of reasoning alignment in Large Reasoning Models. It introduces an online Difficulty Estimator that filters training data before rollouts, ensuring computational resources focus on high‑learning‑potential samples. Empirical results show DEPO achieves about 1.86× faster rollouts than DAPO and outperforms GRPO on benchmarks at similar training costs.
arXiv:2606. 07108v1 Announce Type: new Abstract: Recent advances in Large Reasoning Models (LRMs) demonstrate remarkable performance improvements by iteratively reflecting, exploring, and executing complex tasks, yet suffer from inefficiencies due to redundant reasoning, known as "overthinking".
arXiv:2606. 01281v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs).
The paper argues that in offline preference optimization for reasoning models, applying gradients uniformly to all chosen–rejected pairs is inefficient and can be harmful. It introduces the concept of gradient utility, showing that a pair’s contribution depends on both informativeness and stability, and that high-gradient samples often lie in high‑curvature regions, causing noisy updates. To address this, the authors propose SAGE (Stability‑Aware Gradient Efficiency), which maintains difficulty‑stratified candidate pools and selects only high‑utility pairs for backpropagation, resulting in smoother optimization and better performance on mathematical reasoning benchmarks.
arXiv:2605. 21125v2 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs).
arXiv:2509. 21013v4 Announce Type: replace-cross Abstract: Given the prohibitive cost of pre-training large language models, it is essential to leverage smaller proxy models to optimize datasets before scaling up.
arXiv:2606. 15080v1 Announce Type: cross Abstract: While Large Reasoning Models (LRMs) show strong performance in English, they often fail to reason in the language of the query, a phenomenon known as language collapse.
arXiv:2602. 05547v2 Announce Type: replace-cross Abstract: RL-based post-training with GRPO is widely used to improve large language models on individual reasoning tasks.
The paper introduces Multi-Level Reasoning (MLR), a two-level stochastic framework that separates long chain-of-thought generation into a high-level planner producing structured step descriptors and a low-level executor generating detailed reasoning. MLR replaces single-policy reinforcement learning with an iterative Step‑DPO preference optimization that uses Twisted Sequential Monte Carlo for scalable, stepwise supervision, improving training stability and accuracy. Experiments on math, science, and logic benchmarks show that MLR outperforms both supervised fine‑tuning and strong RL baselines while maintaining robustness over extended reasoning horizons.
The paper surveys efficient reasoning in large language models, contrasting fast intuitive (System 1) and slow deep (System 2) reasoning. It analyzes why System 2 is computationally costly yet more accurate, and why System 1 is efficient but less effective. The survey covers causes of inefficiency, patterns of reasoning behavior, and potential solutions to balance performance and computational budgets, offering actionable insights and an open‑source repository for ongoing research.
arXiv:2606. 07950v1 Announce Type: new Abstract: RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation.
arXiv:2509. 16456v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used in various domains, showing impressive potential on different tasks.
arXiv:2606. 10768v1 Announce Type: new Abstract: The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase.