Improved Regret via Variance-Aware Policy Optimization in Adversarial MDPs
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
arXiv:2602. 09474v2 Announce Type: replace Abstract: We study reinforcement learning in MDPs whose transition function is stochastic at most steps but may behave adversarially at a fixed subset of $\Lambda$ steps per episode.
arXiv:2607. 19854v1 Announce Type: new Abstract: We study horizon-free regret minimization for finite-horizon time-homogeneous tabular Markov decision processes with $S$ states, $A$ actions, horizon $H$, and per-trajectory total reward bounded by $1$.
The paper introduces a novel technique called introduced feedback delays to address regret minimization in episodic Adversarial Markov Decision Processes (AMDPs). By estimating the Q‑function using future episodes while updating the policy, the authors achieve σO(poly(H)√{SAK}) regret in tabular AMDPs and σO(poly(d,H)√{K}) regret in linear AMDPs, resolving long‑standing open problems. The method also extends to delayed AMDPs, providing optimal bounds in terms of state count and total delay.
The paper introduces a technique called introduced feedback delays to improve regret bounds in episodic adversarial Markov Decision Processes (AMDPs). By estimating the Q‑function using future episodes while still updating the policy, the authors achieve µO(poly(H)√{SAK}) regret for tabular AMDPs and µO(poly(d,H)√{K}) regret for linear AMDPs, resolving long‑standing gaps. The method also extends to settings with actual feedback delays, providing optimal bounds in both state and delay dimensions.
The paper addresses policy optimization in online episodic tabular MDPs with adversarial losses and bandit feedback. By employing regularized Q‑functions, the authors eliminate the extra horizon factor present in previous policy‑optimization regret bounds, achieving high‑probability regret of ≥O(√{HS(H+A)T}) for known transitions and ≥O(HS√{AT}) for unknown transitions. These results improve horizon dependence and match the best-known bounds, with extensions to adversarial linear‑mixture MDPs.
arXiv:2610.02520v1 Announce Type: cross Abstract: We study online learning in episodic tabular constrained Markov decision processes with step-wise safety constraints. In such a setting, the constrai...