← Back to all news
arXiv Machine Learning By Han Zhong, Yinyu Ye

Policy Iteration Is Not Strongly Polynomial for Deterministic Markov Decision Processes: The Price of Algorithmic Anarchy

Read the original on arXiv Machine Learning →

The paper presents an exponential lower bound on the number of iterations required by Howard's policy iteration algorithm for deterministic discounted Markov decision processes with at most two actions per state, when the discount factor is part of the input. This result shows that Howard's method cannot be strongly polynomial in this setting and establishes an exponential gap compared to the simplex method with Dantzig's pivoting rule, which remains strongly polynomial. Even with rewards limited to logarithmic bit length, a stretched‑exponential lower bound is achieved, highlighting a fundamental difference between decentralized, simultaneous improvements and Dantzig's coordinated single‑action selection.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
1d ago

Linear Programming Representations and Strongly Polynomial Algorithms for Robust Markov Decision Processes

The paper presents linear programming formulations and strongly polynomial algorithms for robust Markov decision processes (RMDPs) with rational polyhedral state-action rectangular uncertainty in rewards and transitions. By encoding a finite sequence of robust policy-iteration steps, a single LP is constructed whose optimal solutions recover the robust optimal value and all optimal stationary randomized policies. The authors provide a general complexity analysis of robust policy iteration, improving known bounds for α1 and α1∞ RMDPs and establishing new strongly polynomial bounds for general interval, weighted α1, and Wasserstein RMDPs, as well as turn‑based stochastic games with these uncertainty sets.

By Han Zhong, Yinyu Ye
reinforcement-learning
More like this →
arXiv AI
Aug 19

The Curious Case of Exploding DecPOMDPs: Containing the Fire through Policy Counting

The paper introduces policy‑counted DecPOMDPs, a variant of decentralized partially observable Markov decision processes that mitigates the exponential growth in agent numbers by counting policies instead of agents. By exploiting symmetry among agents, the authors achieve a compact representation that reduces model complexity and evaluation cost to polynomial levels. They further present a dynamic programming algorithm that leverages this compact form to solve policy‑counted DecPOMDPs efficiently.

By Nazl{\i} Nur Karabulut, tanya Braun
agentsreinforcement-learning
More like this →
arXiv AI
Jul 10

Provably Optimal Learning Algorithms for Assistance Games

arXiv:2607. 08012v1 Announce Type: cross Abstract: This paper studies an online variant of the assistance games framework, where an informed agent and an uninformed agent repeatedly interact over $T$ timesteps to optimize a common reward function.

By Nivasini Ananthakrishnan, Mark Bedaywi, Michael I. Jordan, Stuart Russell, Nika Haghtalab
agents
More like this →
arXiv Machine Learning
Jun 16

Learning Policy from a Single Trajectory in Average-Reward Markov Decision Process

arXiv:2606. 16729v1 Announce Type: new Abstract: While there is an extensive body of work characterizing the sample complexity of discounted cumulative-reward MDPs, finite sample analyses for average-reward MDPs have been limited, and most existing works rely on restrictive assumptions such as ergodicity or access to a generative model.

By Jongmin Lee, Ernest K. Ryu, Vaneet Aggarwal
reinforcement-learning
More like this →