arXiv Machine Learning

Global Optimality for Constrained Exploration via Penalty Regularization

The paper introduces Policy Gradient Penalty (PGP), a single‑loop policy‑space method that enforces convex occupancy‑measure constraints via quadratic‑penalty regularization. PGP constructs pseudo‑rewards to estimate gradients of the penalized objective and uses the classical Policy Gradient Theorem, establishing smoothness and global last‑iterate convergence guarantees for an ε‑optimal constrained entropy value with ε‑bounded constraint violation. The authors validate PGP with ablations on a grid‑world benchmark and demonstrate scalability on two challenging continuous‑control tasks.

arXiv AI
Jun 6

Retry Policy Gradients in Continuous Action Spaces

arXiv:2606. 05888v1 Announce Type: new Abstract: Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses.

By Soichiro Nishimori, Paavo Parmas
Hugging Face Trending Papers
Nov 5

Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback

The paper introduces Zeroth-Order Log-Barrier Policy Gradient (ZO‑LBPG), a reward‑model‑free method for safe reinforcement learning from human feedback (Safe‑RLHF). ZO‑LBPG optimizes a log‑barrier objective using zeroth‑order estimates derived directly from human preferences, and incorporates conservative constraint margins, coupled rollouts, and an adaptive step size to keep every policy executed during training strictly feasible. The authors prove high‑probability feasibility of all iterates and exploration policies, establish global convergence to an approximately optimal constrained policy, and provide sample complexity bounds for trajectories and human queries, with experimental validation on a synthetic CMDP and a language‑model alignment task.

arXiv Machine Learning
Sep 3

Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learning

The paper introduces Exchange Policy Optimization (EPO), a framework for semi‑infinite safe reinforcement learning that handles infinitely many constraints by iteratively solving finite subproblems. EPO expands or deletes constraints based on tolerance violations and Lagrange multipliers, maintaining computational tractability while converging to an optimal policy with bounded safety violations. The authors prove finite convergence, provide iteration bounds, and quantify the suboptimality gap under mild assumptions.

By Jiaming Zhang, Yujie Yang, Haoning Wang, Liping Zhang, Shengbo Eben Li
arXiv Machine Learning
Nov 6

Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback

The paper introduces Zeroth-Order Log-Barrier Policy Gradient (ZO‑LBPG), a reward‑model‑free method for safe reinforcement learning from human feedback (Safe‑RLHF). ZO‑LBPG optimizes a log‑barrier objective using zeroth‑order estimates from human preferences, ensuring that every policy executed during training remains strictly feasible. The authors prove high‑probability feasibility and global convergence, provide sample complexity bounds, and validate the approach on a synthetic CMDP and a language‑model alignment task.

By Weiqin Chen, Xinmeng Huang, Dongsheng Ding, Santiago Paternain
arXiv Machine Learning
Aug 24

Smart Exploration in Reinforcement Learning using Bounded Uncertainty Models

The paper introduces BUMEX, a reinforcement learning exploration strategy that leverages a set of prior models containing the true transition kernel and reward function. By optimizing over this model set, the method derives upper and lower bounds on the Q‑function to guide exploration, providing theoretical guarantees of convergence to the optimal policy. When the model set follows a bounded‑parameter MDP structure, the optimization becomes convex, enabling finite‑time convergence under mild assumptions and demonstrating accelerated learning in simulations.

By J. S. van Hulst, W. P. M. H. Heemels, D. J. Antunes
arXiv AI
Oct 31

Safe Multi-Objective Reinforcement Learning: Zero Constraint Violation and Primal-Averaging Convergence

The paper introduces two algorithms for safe multi‑objective reinforcement learning that guarantee zero constraint violation during learning. The first algorithm learns a safe Pareto‑optimal policy for a given preference by combining optimistic reward estimates with pessimistic cost bounds, achieving an ×O(1/√T) suboptimality gap. The second, preference‑free algorithm safely explores first and then computes a safe policy for any preference, retaining the same safety guarantee and achieving an ×O(1/√T+1/√K) suboptimality gap.

By Chenxi Li, Yao Shu, Siyang Gao, Shuang Qiu, Guangxin Jiang
arXiv Machine Learning
Oct 8

Convex-Concave Reinforcement Learning

The paper introduces Convex-Concave Reinforcement Learning (CCRL), showing that the exact per‑iteration objective in policy learning can be expressed as a difference‑of‑convex (DC) program in log‑density‑ratio coordinates. This formulation unifies existing methods such as CPI, NPG, TRPO, and AWR as special cases and enables a multi‑step axis that couples consecutive decisions. Using sequential convex programming, the authors provide convergence guarantees and demonstrate that CCRL outperforms or matches PPO on diagnostic MDPs, classic control tasks, and a stochastic mid‑horizon healthcare domain, achieving faster convergence and higher training‑curve area.

By Shripad V. Deshmukh, Yaswanth Chittepu, Dhawal Gupta, Philip Thomas, Scott Niekum