arXiv Machine Learning

Constrained Group Relative Policy Optimization

arXiv Machine Learning
Jun 5

MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following

arXiv:2606. 06058v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards is ideal for multi-constraint instruction following, yet standard group-relative policy optimization (GRPO) becomes unstable under discrete, low-dispersion rewards, where within-group reward distributions are frequently homogeneous.

By Mohammad Mahdi Salmani-Zarchi, Zahra Rahimi, Heshaam Faili, Mohammad Javad Dousti
arXiv AI
Jun 2

Coordination Graphs for Constrained Multi-Agent Reinforcement Learning

arXiv:2606. 02337v1 Announce Type: new Abstract: Constrained Multi-agent reinforcement learning (CMARL) faces two intertwined challenges: the joint action space grows exponentially with the number of agents, and additional requirements couple agents in ways that reward structure alone does not capture.

By Santiago Amaya-Corredor, Miguel Calvo-Fullana, Anders Jonsson
arXiv Machine Learning
5d ago

A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning

The paper presents a unified framework for regularization-based robust reinforcement learning by deriving upper bounds on the performance gap between nominal and worst-case policies. These bounds are expressed as a regularization objective plus a KL-divergence penalty, explaining why KL penalties enhance robustness. The authors reformulate robust training as a constrained optimization problem, updating the Lagrange multiplier jointly with the policy to automatically tune regularization, and validate the approach with extensive adversarial evaluations on continuous control tasks.

By Amine Andam, Jamal Bentahar, Mustapha Hedabou
arXiv Machine Learning
Jun 5

SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Matter

arXiv:2606. 05800v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) often adopts GRPO-style group-relative updates, sampling multiple rollouts per prompt to construct normalized learning signals.

By Powei Chang, Jinpeng Zhang, Chaoqun Sun, MiniWell Tsao, Lianrui Li, Jianxiang Xiang, Chenyu Wang, Yukang Gao, Dongying Kong
arXiv Machine Learning
1d ago

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

The paper introduces QUATRO, a reinforcement‑learning approach for fine‑tuning large language models that enforces trust‑region constraints directly rather than relying on heuristic clipping. By deriving a principled objective, QUATRO provides explicit control over policy updates and stabilizes entropy during training. Experiments on mathematical reasoning benchmarks demonstrate that QUATRO maintains stable training even with higher learning rates and increased policy staleness.

By Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Sookyung Kim, Joonseok Lee, Jaemoo Choi