← Back to all news
arXiv Machine Learning

Optimistic Policy Regularization: Elite-Statistic Reward Shaping

Optimistic Policy Regularization (OPR) is a simple reward‑shaping technique that augments on‑policy agents with a closed‑form bonus derived from elite action priors estimated from top‑quartile episodes. By adding a bounded transform of the log‑ratio of this prior to the policy’s action distribution, OPR requires no extra networks or off‑policy corrections and can be implemented as a one‑line change to PPO’s reward. Experiments on 49 Atari games show that OPR achieves a median human‑normalized score of 148.4 % with only 10 M steps, outperforming published PPO on 36 of 49 games and surpassing self‑imitation learning when matched on frames and backbone.

Related stories

arXiv Machine Learning
Aug 28

Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning

The paper investigates the problem of sharing a single critic across multiple parallel environments in reinforcement learning. It shows that when environments assign different expected returns to the same state, a shared critic must reconcile conflicting value targets, which can distort advantage estimates and misguide policy updates. The authors propose a simple fix—providing the critic with the environment index—demonstrating through bandit models and experiments on CartPole, MuJoCo, BipedalWalker, and 16 Procgen games that this conditional critic stabilizes learning and boosts returns, achieving a 40.8% improvement in aggregate normalized return on unseen levels.

By Zhenya Liu, Yang Meng, Zhuokai Zhao, Xuefeng Liu, Yuxin Chen
reinforcement-learning
More like this →
arXiv AI
Sep 24

ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields

The paper introduces Anchored Neighborhood Optimization (ANO), a new policy‑optimization method that directly designs a smooth, bounded gain field for the probability‑ratio surrogate objective. ANO anchors the identity map at a ratio of one, peaks at a specified trust‑region boundary, and limits the influence of extreme off‑policy samples while providing a bounded, redescending pull on outliers. Empirical results show ANO consistently outperforms existing methods on Atari and MuJoCo benchmarks, and it remains robust under aggressive learning‑rate settings.

By Yiheng Zhang, Yiming Wang, Kaiyan Zhao, Zhenglin Wan, Jiayu Chen, Leong Hou U
llmsreinforcement-learningsafety
More like this →
arXiv Machine Learning
Jul 13

SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions

arXiv:2607. 08925v1 Announce Type: new Abstract: Training reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrained Markov decision process (MDP) formulations do.

By Elham Daneshmand, Majid Khadiv, Glen Berseth, Hsiu-Chin Lin
agentsreinforcement-learningroboticsbenchmarkssafety
More like this →
arXiv AI
Sep 4

FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

arXiv:2609. 03241v1 Announce Type: cross Abstract: A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode.

By Zixun Huang, Kishan Panaganti, Haitao Mi, Leowei Liang
More like this →
arXiv Machine Learning
Oct 8

An Informational Curse of Horizon in Goal-Conditioned Policy Learning

The paper identifies an informational curse of horizon in goal-conditioned policy learning, showing that longer goal relabeling horizons during training can hurt policy generalization and performance. Controlled experiments with oracle planners separate training horizons from test horizons, revealing that behavioral cloning policies degrade with longer horizons while reinforcement learning mitigates this effect. The authors attribute this to reduced conditional mutual information between actions and hindsight-relabeled goals, and demonstrate that distilling short-horizon policy Jacobians into long-horizon policies improves performance, especially in combinatorial manipulation tasks.

By John L. Zhou, Yuxuan Dong, Jonathan C. Kao
reinforcement-learningroboticssafety
More like this →