arXiv Machine Learning By Kaixuan Liu, Guojun Xiong, Shengpu Tang, Wanyun Si, Jian Li

Provably Efficient Representation Learning for Low-Rank CMDPs

Read the original on arXiv Machine Learning →

The paper introduces “MFRLC”, a model‑free algorithm for representation learning in low‑rank Constrained Markov Decision Processes (CMDPs). It combines Least‑Squares Value Iteration with Upper Confidence Bound and a primal‑dual scheme to balance reward maximization and constraint compliance, learning value functions directly from representations without estimating transition kernels. The authors prove that “MFRLC” achieves sublinear “O(K^{3/4})” reward regret and cumulative constraint violation over K rounds, independent of the state‑space size.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
Oct 23

Primal-Dual Representation Learning for Low-Rank Constrained MDPs

The paper introduces REP-PD-TEG, a representation‑learning algorithm for low‑rank continuous‑state constrained Markov Decision Processes (CMDPs) that achieves optimal ≥O(√K) regret in the soft cumulative‑violation setting while maintaining a ≥O(K^{3/4}) sample bound when exploratory actions are penalised. For the stricter hard cumulative‑violation scenario, REP-PD-H-TEG is proposed, which uses a bonus‑corrected lower‑confidence utility certificate and a stabilized dual search to guarantee both regret and hard‑violation bounds. Together, these methods provide the first theoretically guaranteed representation‑learning framework for low‑rank continuous‑state CMDPs with both soft and hard violation controls.

By Chenhao Zhou, Chao Zhang, Wensong Bai, Hanbin Zhao, Hui Qian
arXiv Machine Learning
Oct 3

Rate-Optimal Algorithm for Adversarial Linear CMDPs

The paper introduces a new primal–dual algorithm for episodic adversarial linear constrained Markov decision processes (CMDPs) with unknown transitions. It achieves a rate‑optimal ×O(√K) regret and cumulative constraint violation, improving upon the previous ×O(K^{3/4}) bound and eliminating the need for Slater’s condition. The method combines adaptive FTRL, contracted value estimation, and an exponential Lyapunov function, enabling uniform concentration over the value function class and computational efficiency independent of the state‑space size.

By Kihyun Yu, Honghao Wei, Dabeen Lee
arXiv Machine Learning
Oct 2

Learning Infinite-Horizon Average-Reward CMDPs via State Augmentation

The paper introduces a computationally efficient algorithm for infinite-horizon average-reward constrained Markov decision processes (CMDPs) under weak communication. It achieves a high-probability regret and cumulative constraint violation of ×O(√T) in the tabular setting, matching optimal dependence up to logarithmic factors. The method augments the state with cumulative constraint violation, reshapes rewards using a Huber potential, and applies finite-horizon approximation with optimistic value iteration to maintain bounded per-step rewards.

By Kihyun Yu, Seoungbin Bae, Dabeen Lee