The paper presents an exponential lower bound on the number of iterations required by Howard's policy iteration algorithm for deterministic discounted Markov decision processes with at most two actions per state, when the discount factor is part of the input. This result shows that Howard's method cannot be strongly polynomial in this setting and establishes an exponential gap compared to the simplex method with Dantzig's pivoting rule, which remains strongly polynomial. Even with rewards limited to logarithmic bit length, a stretched‑exponential lower bound is achieved, highlighting a fundamental difference between decentralized, simultaneous improvements and Dantzig's coordinated single‑action selection.
arXiv:2609.00504v1 Announce Type: cross Abstract: In this work, we study radically uncoupled learning in discounted general-sum Markov games. Assuming ``$\mathsf{ETH}$ for $\mathsf{PPAD}$", we show t...
arXiv:2601. 23229v2 Announce Type: replace Abstract: Markov decision processes (MDPs) are a fundamental model in sequential decision making.
The paper presents linear programming formulations and strongly polynomial algorithms for robust Markov decision processes (RMDPs) with rational polyhedral state-action rectangular uncertainty in rewards and transitions. By encoding a finite sequence of robust policy-iteration steps, a single LP is constructed whose optimal solutions recover the robust optimal value and all optimal stationary randomized policies. The authors provide a general complexity analysis of robust policy iteration, improving known bounds for α1 and α1∞ RMDPs and establishing new strongly polynomial bounds for general interval, weighted α1, and Wasserstein RMDPs, as well as turn‑based stochastic games with these uncertainty sets.
The paper introduces policy‑counted DecPOMDPs, a variant of decentralized partially observable Markov decision processes that mitigates the exponential growth in agent numbers by counting policies instead of agents. By exploiting symmetry among agents, the authors achieve a compact representation that reduces model complexity and evaluation cost to polynomial levels. They further present a dynamic programming algorithm that leverages this compact form to solve policy‑counted DecPOMDPs efficiently.
arXiv:2607. 08012v1 Announce Type: cross Abstract: This paper studies an online variant of the assistance games framework, where an informed agent and an uninformed agent repeatedly interact over $T$ timesteps to optimize a common reward function.
arXiv:2606. 16729v1 Announce Type: new Abstract: While there is an extensive body of work characterizing the sample complexity of discounted cumulative-reward MDPs, finite sample analyses for average-reward MDPs have been limited, and most existing works rely on restrictive assumptions such as ergodicity or access to a generative model.
arXiv:2606. 27448v1 Announce Type: new Abstract: This paper studies the problem of regret minimization in Markovian bandits with \emph{non-observable states} and possibly \emph{constrained} decision epochs.
We study infinite-horizon average-reward constrained Markov decision processes (CMDPs) under the weakly communicating assumption. Existing high-probability guarantees for this setting either require c...
The paper introduces the categorizer automaton, a deterministic automaton that processes an infinite sequence of rewards and determines which of a finite set of bins contains the discounted sum. Unlike previous approaches that combine multiple comparator automata and yield exponential state spaces, the authors construct a categorizer automaton with a state space linear in the number of bins. They apply this construction to Markov decision processes, enabling synthesis of policies that maximize expected utility for discounted-sum payoffs, including cases with discontinuous or piecewise‑Lipschitz utility functions, achieving pseudo‑polynomial time algorithms and proving PSPACE‑hardness for the synthesis problem even with piecewise‑constant utilities.
arXiv:2609.38132v1 Announce Type: new Abstract: We study average-reward weakly-coupled Markov decision processes (WCMDPs), where a WCMDP consists of $N$ smaller MDPs, called arms, that share multiple...
The paper introduces a framework for optimal policy improvement in reinforcement learning, defining it as the best single update under given constraints. It shows that restricting improvement to a subset of states is equivalent to solving an induced Markov Decision Process, linking planning with explicit or implicit models to optimal policy improvement. The authors develop a novel operator for greedification under approximate evaluation, demonstrating empirical gains across several RL algorithms and settings.
arXiv:2606. 14095v1 Announce Type: new Abstract: We study the sample complexity of learning in average-reward weakly-coupled Markov decision processes (WCMDPs) and Restless Bandits (RBs) under a generative model.