arXiv AI October 28, 2026 By Yu Zhao, Fan Jiang, Tianle Liu, Bo Zeng, Xintong Wang, Jiaxuan Teng, Yu Liu, Yichao Du

Difficulty-Estimated Policy Optimization

Difficulty-Estimated Policy Optimization (DEPO) is a new framework that improves the efficiency and robustness of reasoning alignment in Large Reasoning Models. It introduces an online Difficulty Estimator that filters training data before rollouts, ensuring computational resources focus on high‑learning‑potential samples. Empirical results show DEPO achieves about 1.86× faster rollouts than DAPO and outperforms GRPO on benchmarks at similar training costs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 8

DyCon: Dynamic Reasoning Control via Evolving Difficulty Modeling

arXiv:2606. 07108v1 Announce Type: new Abstract: Recent advances in Large Reasoning Models (LRMs) demonstrate remarkable performance improvements by iteratively reflecting, exploring, and executing complex tasks, yet suffer from inefficiencies due to redundant reasoning, known as "overthinking".

By Tengyao Tu, Yulin Li, Hui-Ling Zhen, Libo Qin, Zhoujun Wei, Jinghua Piao, Zhuotao Tian, Yong Li, Min Zhang

rag nlp benchmarks

arXiv AI

Jun 2

RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

arXiv:2606. 01281v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs).

By Yixiu Mao, Yun Qu, Qi Wang, Heming Zou, Xiangyang Ji

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Sep 4

Not All Preferences Deserve Gradients: Understanding Gradient Utility in Offline Reasoning Alignment

The paper argues that in offline preference optimization for reasoning models, applying gradients uniformly to all chosen–rejected pairs is inefficient and can be harmful. It introduces the concept of gradient utility, showing that a pair’s contribution depends on both informativeness and stability, and that high-gradient samples often lie in high‑curvature regions, causing noisy updates. To address this, the authors propose SAGE (Stability‑Aware Gradient Efficiency), which maintains difficulty‑stratified candidate pools and selects only high‑utility pairs for backpropagation, resulting in smoother optimization and better performance on mathematical reasoning benchmarks.

By Hui Wu, Hengyi Cai, Jinman Zhao, Xinran Chen, Ziheng Li, Zhejun Zhao, Shuaiqiang Wang, Yuchen Li, Dawei Yin

benchmarks safety

arXiv Machine Learning

Jun 2

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

arXiv:2605. 21125v2 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs).

By Xixiang He, Qiyao Sun, Ao Cheng, Xingming Li, Xuanyu Ji, Hailun Lu, Runke Huang, Qingyong Hu

llms reinforcement-learning benchmarks

arXiv AI

Jul 2

Predicting LLM Reasoning Performance with Small Proxy Model

arXiv:2509. 21013v4 Announce Type: replace-cross Abstract: Given the prohibitive cost of pre-training large language models, it is essential to leverage smaller proxy models to optimize datasets before scaling up.

By Woosung Koh, Juyoung Suk, Sungjun Han, Se-Young Yun, Jamin Shin

llms benchmarks safety

arXiv AI

Jun 16

AdaMame: A Training Recipe for Adaptive Multilingual Reasoning

arXiv:2606. 15080v1 Announce Type: cross Abstract: While Large Reasoning Models (LRMs) show strong performance in English, they often fail to reason in the language of the query, a phenomenon known as language collapse.

By Dayeon Ki, Kevin Duh, Marine Carpuat

reinforcement-learning fine-tuning benchmarks safety

One email a morning, machine-written

Related stories

DyCon: Dynamic Reasoning Control via Evolving Difficulty Modeling

RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

Not All Preferences Deserve Gradients: Understanding Gradient Utility in Offline Reasoning Alignment

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

Predicting LLM Reasoning Performance with Small Proxy Model

AdaMame: A Training Recipe for Adaptive Multilingual Reasoning