← Back to all news
arXiv Machine Learning By Mai Pham, Vikrant Vaze, Peter Chin

Optimistic Policy Regularization: Elite-Statistic Reward Shaping

Read the original on arXiv Machine Learning →

Optimistic Policy Regularization (OPR) is a simple reward‑shaping technique that augments on‑policy agents with a closed‑form bonus derived from elite action priors estimated from top‑quartile episodes. By adding a bounded transform of the log‑ratio of this prior to the policy’s action distribution, OPR requires no extra networks or off‑policy corrections and can be implemented as a one‑line change to PPO’s reward. Experiments on 49 Atari games show that OPR achieves a median human‑normalized score of 148.4 % with only 10 M steps, outperforming published PPO on 36 of 49 games and surpassing self‑imitation learning when matched on frames and backbone.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 28

Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning

The paper investigates the problem of sharing a single critic across multiple parallel environments in reinforcement learning. It shows that when environments assign different expected returns to the same state, a shared critic must reconcile conflicting value targets, which can distort advantage estimates and misguide policy updates. The authors propose a simple fix—providing the critic with the environment index—demonstrating through bandit models and experiments on CartPole, MuJoCo, BipedalWalker, and 16 Procgen games that this conditional critic stabilizes learning and boosts returns, achieving a 40.8% improvement in aggregate normalized return on unseen levels.

By Zhenya Liu, Yang Meng, Zhuokai Zhao, Xuefeng Liu, Yuxin Chen
reinforcement-learning
More like this →
arXiv AI
Sep 24

ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields

The paper introduces Anchored Neighborhood Optimization (ANO), a new policy‑optimization method that directly designs a smooth, bounded gain field for the probability‑ratio surrogate objective. ANO anchors the identity map at a ratio of one, peaks at a specified trust‑region boundary, and limits the influence of extreme off‑policy samples while providing a bounded, redescending pull on outliers. Empirical results show ANO consistently outperforms existing methods on Atari and MuJoCo benchmarks, and it remains robust under aggressive learning‑rate settings.

By Yiheng Zhang, Yiming Wang, Kaiyan Zhao, Zhenglin Wan, Jiayu Chen, Leong Hou U
llmsreinforcement-learningsafety
More like this →