arXiv:2607. 08925v1 Announce Type: new Abstract: Training reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrained Markov decision process (MDP) formulations do.
By Elham Daneshmand, Majid Khadiv, Glen Berseth, Hsiu-Chin Lin
arXiv:2511. 02577v2 Announce Type: replace Abstract: Proximal Policy Optimization (PPO) is widely regarded as one of the most successful deep reinforcement learning algorithms, known for its robustness and effectiveness across a range of problems.
By Gilad Karpel, Ruida Zhou, Shoham Sabach, Mohammad Ghavamzadeh
arXiv:2607.08925v2 Announce Type: replace
Abstract: Training agents directly on the task they will be deployed on often leads to the best policy performance, but training on physical robots can cause...
By Elham Daneshmand, Majid Khadiv, Glen Berseth, Hsiu-Chin Lin
arXiv:2607. 02206v1 Announce Type: cross Abstract: Predictions are increasingly used to guide high-stakes decisions, from treatment selection to policy making.
By Yurui Zheng, Ying Jin
The paper investigates how learned visual reward models can inadvertently encourage robot policies to perform poorly on the intended task while still receiving high reward signals. By fine‑tuning a diffusion policy on a drawer‑opening task using a learned reward, the authors observe that task success increases but so does the frequency of wrong‑object failures, a phenomenon that also appears when the policy is re‑optimized with the same reward. A tilt model explains that outcomes with higher initial expected reward become more frequent under KL‑regularized optimization, and a separate outcome verifier can redirect the policy toward the correct task.
By Jiaxuan Luo, Xingguo Xu, Shanshan Wang, Yuhan Zhou, Zhen Zhang
The paper introduces Generalized Proximal Policy Optimization (GPPO), a unified framework that extends PPO by incorporating $f$-divergence trust-region theory and novel estimator topologies for large language model reinforcement learning. GPPO demonstrates that different divergences produce distinct policy update dynamics and that estimator design significantly impacts performance. Experiments on eight math and coding benchmarks show GPPO surpasses standard GRPO by up to 50% in math and 14% in coding, achieving state‑of‑the‑art results.
By Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Weinan Dai, Yuyan Zhou, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
arXiv:2608. 01130v1 Announce Type: new Abstract: A broad range of models face the mismatch where they are updated through trajectory losses but are evaluated by downstream task reward.
By Yuyang Shen
The paper investigates whether the policy gradient update used in many reinforcement learning algorithms—specifically the episode-summed, on‑policy update that omits the outer discount factor—can be interpreted as the gradient of any objective function. The authors prove that this update does not necessarily correspond to the gradient of any function and provide a counterexample where the update drives the policy toward a globally pessimal deterministic limit while the parameters diverge. They also highlight that this theoretical ambiguity has led to widespread misstatements in the literature, even in highly cited works from top conferences.
By Chris Nota, Philip S. Thomas
arXiv:2602. 09474v2 Announce Type: replace Abstract: We study reinforcement learning in MDPs whose transition function is stochastic at most steps but may behave adversarially at a fixed subset of $\Lambda$ steps per episode.
By Ofir Schlisselberg, Tal Lancewicki, Yishay Mansour
arXiv:2602. 10430v2 Announce Type: replace-cross Abstract: Off-policy policy optimization reuses historical behavior, including negative-advantage samples that suppress known failures.
By Yusen Huo, Changping Wang, Yangru Huang, Jun Zhang, Jie Jiang
arXiv:2512. 23075v5 Announce Type: replace-cross Abstract: Policy gradient methods for Large Language Models optimize a policy $\pi_\theta$ via a surrogate objective computed from samples of a rollout policy $\pi_{\text{roll}}$.
By Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Qian Liu, Baoxiang Wang
The paper identifies an informational curse of horizon in goal-conditioned policy learning, showing that longer goal relabeling horizons during training can hurt policy generalization and performance. Controlled experiments with oracle planners separate training horizons from test horizons, revealing that behavioral cloning policies degrade with longer horizons while reinforcement learning mitigates this effect. The authors attribute this to reduced conditional mutual information between actions and hindsight-relabeled goals, and demonstrate that distilling short-horizon policy Jacobians into long-horizon policies improves performance, especially in combinatorial manipulation tasks.
By John L. Zhou, Yuxuan Dong, Jonathan C. Kao