The paper introduces the categorizer automaton, a deterministic automaton that processes an infinite sequence of rewards and determines which of a finite set of bins contains the discounted sum. Unlike previous approaches that combine multiple comparator automata and yield exponential state spaces, the authors construct a categorizer automaton with a state space linear in the number of bins. They apply this construction to Markov decision processes, enabling synthesis of policies that maximize expected utility for discounted-sum payoffs, including cases with discontinuous or piecewise‑Lipschitz utility functions, achieving pseudo‑polynomial time algorithms and proving PSPACE‑hardness for the synthesis problem even with piecewise‑constant utilities.
arXiv:2511. 19849v2 Announce Type: replace-cross Abstract: Recurrence objectives, where a target region must be visited infinitely often, are a fundamental class of specifications for Markov decision processes (MDPs) and form the core of $\omega$-regular and linear temporal logic (LTL) objectives.
The paper presents an exponential lower bound on the number of iterations required by Howard's policy iteration algorithm for deterministic discounted Markov decision processes with at most two actions per state, when the discount factor is part of the input. This result shows that Howard's method cannot be strongly polynomial in this setting and establishes an exponential gap compared to the simplex method with Dantzig's pivoting rule, which remains strongly polynomial. Even with rewards limited to logarithmic bit length, a stretched‑exponential lower bound is achieved, highlighting a fundamental difference between decentralized, simultaneous improvements and Dantzig's coordinated single‑action selection.
We study infinite-horizon average-reward constrained Markov decision processes (CMDPs) under the weakly communicating assumption. Existing high-probability guarantees for this setting either require c...
arXiv:2603. 23461v2 Announce Type: replace Abstract: We study reinforcement learning (RL) with linear function approximation in Markov Decision Processes (MDPs) satisfying \emph{linear Bellman completeness} -- a fundamental setting where the Bellman backup of any linear value function remains linear.
arXiv:2601. 23229v2 Announce Type: replace Abstract: Markov decision processes (MDPs) are a fundamental model in sequential decision making.
The paper investigates adaptive policy portfolios for Robust Markov Decision Processes (RMDPs), proposing finite sets of memoryless randomized policies generated offline and selected online. It introduces robust regret as a metric for portfolio quality, comparing each portfolio member’s performance to the optimal policy for each plausible environment. The authors provide complexity-theoretic results showing that certifying and synthesizing such portfolios is highly intractable, and they present an offline construction method that can be specialized at runtime.
arXiv:2609.39093v1 Announce Type: new Abstract: We study infinite-horizon average-reward constrained Markov decision processes (CMDPs) under the weakly communicating assumption. Existing high-probabi...
The paper introduces a new approach to learning chance-constrained Markov decision processes (CCMDPs) using a Bellman distributional certificate. It provides both model-based and model-free algorithms with theoretical guarantees, including matching upper and lower bounds for tabular discounted CCMDPs with bounded successor support. Numerical experiments on synthetic CCMDPs and an IEEE 14-bus energy storage benchmark demonstrate the safety and effectiveness of the proposed methods.
The paper introduces adaptive policy portfolios for robust Markov decision processes, where a finite set of memoryless randomized policies is synthesized offline and paired with an online selector. It defines robust regret as a measure of portfolio quality, comparing each portfolio member to the optimal policy for each plausible environment. The authors provide a complexity-theoretic analysis of portfolio certification and synthesis, showing that even deterministic portfolios in simple settings are highly complex, and present an offline construction method that can be specialized at runtime.
arXiv:2602. 03778v2 Announce Type: replace-cross Abstract: Tail-end risk measures such as static conditional value-at-risk (CVaR) are used in safety-critical applications to prevent rare, yet catastrophic events.
arXiv:2608. 13625v1 Announce Type: new Abstract: Signal temporal logic (STL) provides a formal language for specifying real-time properties of real-valued observations, along with a quantitative robustness score for monitoring satisfaction.
arXiv:2607. 15459v1 Announce Type: new Abstract: A trained deep reinforcement learning policy is a black box, and we ask whether it can be made explainable by rewriting it as an executable logic program that reproduces its behaviour and that a person can read, a logic engine can run, and an optimizer can edit.