arXiv Machine Learning By Florian Wolf, Ilyas Fatkhullin, Niao He

Global Optimality for Constrained Exploration via Penalty Regularization

Read the original on arXiv Machine Learning →

The paper introduces Policy Gradient Penalty (PGP), a single‑loop policy‑space method that enforces convex occupancy‑measure constraints via quadratic‑penalty regularization. PGP constructs pseudo‑rewards to estimate gradients of the penalized objective and uses the classical Policy Gradient Theorem, establishing smoothness and global last‑iterate convergence guarantees for an ε‑optimal constrained entropy value with ε‑bounded constraint violation. The authors validate PGP with ablations on a grid‑world benchmark and demonstrate scalability on two challenging continuous‑control tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Jun 6

Retry Policy Gradients in Continuous Action Spaces

arXiv:2606. 05888v1 Announce Type: new Abstract: Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses.

By Soichiro Nishimori, Paavo Parmas
Hugging Face Trending Papers
Nov 5

Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback

The paper introduces Zeroth-Order Log-Barrier Policy Gradient (ZO‑LBPG), a reward‑model‑free method for safe reinforcement learning from human feedback (Safe‑RLHF). ZO‑LBPG optimizes a log‑barrier objective using zeroth‑order estimates derived directly from human preferences, and incorporates conservative constraint margins, coupled rollouts, and an adaptive step size to keep every policy executed during training strictly feasible. The authors prove high‑probability feasibility of all iterates and exploration policies, establish global convergence to an approximately optimal constrained policy, and provide sample complexity bounds for trajectories and human queries, with experimental validation on a synthetic CMDP and a language‑model alignment task.

arXiv Machine Learning
Sep 3

Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learning

The paper introduces Exchange Policy Optimization (EPO), a framework for semi‑infinite safe reinforcement learning that handles infinitely many constraints by iteratively solving finite subproblems. EPO expands or deletes constraints based on tolerance violations and Lagrange multipliers, maintaining computational tractability while converging to an optimal policy with bounded safety violations. The authors prove finite convergence, provide iteration bounds, and quantify the suboptimality gap under mild assumptions.

By Jiaming Zhang, Yujie Yang, Haoning Wang, Liping Zhang, Shengbo Eben Li
arXiv Machine Learning
Nov 6

Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback

The paper introduces Zeroth-Order Log-Barrier Policy Gradient (ZO‑LBPG), a reward‑model‑free method for safe reinforcement learning from human feedback (Safe‑RLHF). ZO‑LBPG optimizes a log‑barrier objective using zeroth‑order estimates from human preferences, ensuring that every policy executed during training remains strictly feasible. The authors prove high‑probability feasibility and global convergence, provide sample complexity bounds, and validate the approach on a synthetic CMDP and a language‑model alignment task.

By Weiqin Chen, Xinmeng Huang, Dongsheng Ding, Santiago Paternain