← Back to all news
arXiv Machine Learning By Maxime Baelde

Parametric Resonance in Primal-Dual Learning for Slowly Non-Stationary Constrained MDPs

Read the original on arXiv Machine Learning →

The paper analyzes primal‑dual algorithms for constrained Markov decision processes under a two‑timescale framework, focusing on how non‑stationarity enters via additive and parametric channels. In the additive case, oscillating constraint levels cause the dual variable to saturate at a floor determined by the moving optimum, with amplification proportional to the timescale ratio. In the parametric case, oscillating constraint sensitivity leads to a parametric resonance at twice the natural frequency, driven by mismatches between primal and dual estimates; this resonance is mitigated by aligning the updates or using a PID term.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 3

Rate-Optimal Algorithm for Adversarial Linear CMDPs

The paper introduces a new primal–dual algorithm for episodic adversarial linear constrained Markov decision processes (CMDPs) with unknown transitions. It achieves a rate‑optimal ×O(√K) regret and cumulative constraint violation, improving upon the previous ×O(K^{3/4}) bound and eliminating the need for Slater’s condition. The method combines adaptive FTRL, contracted value estimation, and an exponential Lyapunov function, enabling uniform concentration over the value function class and computational efficiency independent of the state‑space size.

By Kihyun Yu, Honghao Wei, Dabeen Lee
reinforcement-learningsafety
More like this →
arXiv AI
Sep 24

ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields

The paper introduces Anchored Neighborhood Optimization (ANO), a new policy‑optimization method that directly designs a smooth, bounded gain field for the probability‑ratio surrogate objective. ANO anchors the identity map at a ratio of one, peaks at a specified trust‑region boundary, and limits the influence of extreme off‑policy samples while providing a bounded, redescending pull on outliers. Empirical results show ANO consistently outperforms existing methods on Atari and MuJoCo benchmarks, and it remains robust under aggressive learning‑rate settings.

By Yiheng Zhang, Yiming Wang, Kaiyan Zhao, Zhenglin Wan, Jiayu Chen, Leong Hou U
llmsreinforcement-learningsafety
More like this →
arXiv Machine Learning
Jun 11

Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity

arXiv:2606. 11431v1 Announce Type: new Abstract: Mirror Descent (MD) extends Gradient Descent (GD) beyond Euclidean geometry and has recently reappeared as a lens for KL-regularized policy optimization in reinforcement learning and LLM post-training.

By Shira Vansover-Hager, Matan Schliserman, Ofir Schlisselberg, Tomer Koren
llmsreinforcement-learning
More like this →