arXiv AI By Aarush Kulkarni, Khang Nguyen, William Chang, Yan Dai

Introducing Feedback Delays to Non-Delayed Adversarial MDPs: Optimal $\sqrt S$ and $\sqrt K$ Dependencies in Tabular and Linear AMDPs

Read the original on arXiv AI →

The paper introduces a technique called introduced feedback delays to improve regret bounds in episodic adversarial Markov Decision Processes (AMDPs). By estimating the Q‑function using future episodes while still updating the policy, the authors achieve µO(poly(H)√{SAK}) regret for tabular AMDPs and µO(poly(d,H)√{K}) regret for linear AMDPs, resolving long‑standing gaps. The method also extends to settings with actual feedback delays, providing optimal bounds in both state and delay dimensions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers
4d ago

Introducing Feedback Delays to Non-Delayed Adversarial MDPs: Optimal $\sqrt S$ and $\sqrt K$ Dependencies in Tabular and Linear AMDPs

The paper introduces a novel technique called introduced feedback delays to address regret minimization in episodic Adversarial Markov Decision Processes (AMDPs). By estimating the Q‑function using future episodes while updating the policy, the authors achieve σO(poly(H)√{SAK}) regret in tabular AMDPs and σO(poly(d,H)√{K}) regret in linear AMDPs, resolving long‑standing open problems. The method also extends to delayed AMDPs, providing optimal bounds in terms of state count and total delay.

arXiv Statistics ML
Oct 26

From Dueling Bandits to Episodic MDPs: Learning with Adversarial Preference Feedback

The paper extends dueling bandits to episodic Markov decision processes where selecting a pair of arms not only yields a noisy preference but also influences the next state. It defines a state‑wise preference‑based MDP with adversarially changing preferences and Borda‑score losses, and proves a minimax regret of ∫Θ(min{HT,(H^3SK)^{1/3}T^{2/3}}) under known transitions. The authors provide an online linear‑optimization algorithm achieving this bound, a policy‑optimization algorithm with closed‑form updates, and extensions to unknown transitions.

By Taira Tsuchiya, Shinji Ito, Haipeng Luo