← Back to all news
arXiv Statistics ML By Taira Tsuchiya, Shinji Ito, Haipeng Luo

From Dueling Bandits to Episodic MDPs: Learning with Adversarial Preference Feedback

Read the original on arXiv Statistics ML →

The paper extends dueling bandits to episodic Markov decision processes where selecting a pair of arms not only yields a noisy preference but also influences the next state. It defines a state‑wise preference‑based MDP with adversarially changing preferences and Borda‑score losses, and proves a minimax regret of ∫Θ(min{HT,(H^3SK)^{1/3}T^{2/3}}) under known transitions. The authors provide an online linear‑optimization algorithm achieving this bound, a policy‑optimization algorithm with closed‑form updates, and extensions to unknown transitions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Statistics ML.

Related stories

arXiv Machine Learning
Oct 9

Closing the Horizon Gap in Policy Optimization for Adversarial MDPs

The paper addresses policy optimization in online episodic tabular MDPs with adversarial losses and bandit feedback. By employing regularized Q‑functions, the authors eliminate the extra horizon factor present in previous policy‑optimization regret bounds, achieving high‑probability regret of ≥O(√{HS(H+A)T}) for known transitions and ≥O(HS√{AT}) for unknown transitions. These results improve horizon dependence and match the best-known bounds, with extensions to adversarial linear‑mixture MDPs.

By Mingyi Li, Taira Tsuchiya
reinforcement-learningsafety
More like this →
arXiv AI
Oct 16

Replicable Dueling Bandits

The paper introduces the first algorithms for replicable dueling bandits, ensuring that two runs with shared internal randomness and independent comparison outcomes select the same ordered pair in every round with high probability. Under relaxed stochastic transitivity, the authors achieve expected regret ×O(γ^8 K^4 H_p/ρ^2), while under strong stochastic transitivity they improve this to ×O(K^2 H_p/ρ^2). For linear dueling bandits, a regret bound of ×O(d^3√T/(c_μρ)) is presented, and experiments confirm both replicability and sublinear regret.

By Rohan Deb, Aadirupa Saha, Udaya Ghai, Karan Singh, Arindam Banerjee
More like this →