arXiv AI

Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models

The paper proposes a two‑channel constrained GRPO framework that treats safety as explicit budgets on harmful assistance and over‑refusal rates, using Lagrange multipliers and a monotone mapping of scorer outputs. It demonstrates that this calibration reduces harmful responses from 8.9% to 4.8% on a Qwen3‑VL‑2B policy while keeping over‑refusal low, outperforming standard GRPO and uncalibrated budgets across multiple benchmarks.

arXiv AI
6d ago

Are Dense Rewards Worth Their Tokens in GRPO?

The paper evaluates whether dense, token‑level rewards improve training of language models compared to the standard group‑relative policy optimization (GRPO) algorithm. Using a matched‑compute protocol that equalizes generated tokens and GPU‑hours across interventions, the authors find that dense rewards—such as Monte‑Carlo step‑wise scoring—do not outperform GRPO, and in some cases perform worse even when matched for compute. Instead, a simple strategy of training on prompts predicted to be near the 50% success threshold yields consistent accuracy gains across multiple models and settings.

By Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri, Sreehari Sankar, Wang Yang, Chaoda Song, Alexander Nemecek, Osama Zafar, Yuting Shao, Kai Ye, Biyao Zhang, Xinpeng Li, Shouren Wang, Erman Ayday, Jing Ma, Sanmukh Rao Kuppannagari, Vipin Chaudhary
arXiv AI
Jul 15

A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

arXiv:2607. 12640v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization (GRPO) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent.

By Chengguang Gan, Zhixi Cai, Yunhao Liang, Hanjun Wei, Shiwen Ni, Qinghao Zhang
arXiv AI
Oct 12

Unsupervised Multimodal Reward Modeling

The paper presents an unsupervised multimodal reward model built on a frozen Qwen3‑VL backbone. By adding a linear head to an intermediate layer and training it to rank a web caption higher than a perturbed version of itself, the authors avoid human preference labels and achieve competitive performance—outperforming GPT‑4o on several benchmarks with far fewer parameters and compute. The approach demonstrates that a carefully chosen intermediate layer, simple perturbation-based training, and verifier‑driven fine‑tuning can yield effective vision‑language rewards without costly annotation.

By Yueying Li, Jingxuan Fan, Kiant\'e Brantley, Sham M. Kakade, Hanlin Zhang
arXiv AI
6d ago

Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards

The paper introduces MCPO (Margin-Credit Policy Optimization), a method that allocates reinforcement learning rewards to tokens where a language model hesitates, measured by the top‑2 margin of the next‑token distribution. MCPO redistributes each solution’s advantage across its tokens while keeping the mean unchanged, requiring no extra model evaluations. Experiments on MiniCPM5‑2B‑Base and Qwen3‑4B‑Base show MCPO consistently outperforms the baseline GRPO across multiple math benchmarks, especially on harder problems, and encourages longer, higher‑scoring answers.

By Chi Liu, Xin Chen, Yuchuan Weng, Srinivasan Manoharan
arXiv AI
Aug 13

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

arXiv:2608. 11669v1 Announce Type: cross Abstract: Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer.

By Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Ying Liu