arXiv Machine Learning
Aug 11

Directional-Clamp PPO

arXiv:2511. 02577v2 Announce Type: replace Abstract: Proximal Policy Optimization (PPO) is widely regarded as one of the most successful deep reinforcement learning algorithms, known for its robustness and effectiveness across a range of problems.

By Gilad Karpel, Ruida Zhou, Shoham Sabach, Mohammad Ghavamzadeh
arXiv Machine Learning
Jul 13

SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions

arXiv:2607. 08925v1 Announce Type: new Abstract: Training reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrained Markov decision process (MDP) formulations do.

By Elham Daneshmand, Majid Khadiv, Glen Berseth, Hsiu-Chin Lin
arXiv Computation and Language
Oct 1

OPTS-TTPO: Enhancing Finite-Sample Policy-Gradient Learning with Tree Search

arXiv:2609.40035v1 Announce Type: new Abstract: The policy-gradient theorem gives the exact gradient under the current policy, but finite on-policy samples may miss rare high-return trajectories. We...

By Junyu Lu, Shichao Weng, Zhiqiang Wang, Haojie Luo, Jingfan Zhang, Yuhua Zhou, Cheng Du, Yuzhuo Zhang, Xi Li, Jinwei Du, Tiancheng Feng, Chuan Xiao, Shuyuan Zheng
arXiv Machine Learning
Oct 3

Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?

The paper investigates how reusing past samples can improve the sample efficiency of Proximal Policy Optimization (PPO). Two variants, wPPO-U and wPPO-BH, are introduced within a multiple importance weighting framework, each reusing data from recent iterations while preserving core PPO mechanics. The authors derive theoretical policy improvement bounds for both variants and empirically evaluate their impact on continuous control tasks.

By Alessandro Montenegro, Riccardo Venturelli, Marco Mussi, Matteo Papini, Alberto Maria Metelli
arXiv AI
3d ago

CARE-PO: Controllable Tool-Call Efficiency for LLM Agents

The paper introduces CARE-PO, a method for controlling tool‑call efficiency in large language model agents. By anchoring an efficiency reward to a fixed reference budget tied to task difficulty and injecting successful trajectories into training rollouts, CARE‑PO prevents the collapse to zero tool calls and allows a single training run to produce multiple operating points balancing efficiency and capability. Experiments on the GAIA benchmark and other multi‑hop tasks show that CARE‑PO improves both success rates and call counts across several model sizes.

By Xiaoyang Ye, Xiao Yao, Hairui Zhu, Yiying Yang, Bei Gan, Jinliang Lu, Tengjin Weng, Yuan Yao, Wenhao Jiang