arXiv AI By Bo Han

Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models

Read the original on arXiv AI →

The paper proposes a two‑channel constrained GRPO framework that treats safety as explicit budgets on harmful assistance and over‑refusal rates, using Lagrange multipliers and a monotone mapping of scorer outputs. It demonstrates that this calibration reduces harmful responses from 8.9% to 4.8% on a Qwen3‑VL‑2B policy while keeping over‑refusal low, outperforming standard GRPO and uncalibrated budgets across multiple benchmarks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
6d ago

Are Dense Rewards Worth Their Tokens in GRPO?

The paper evaluates whether dense, token‑level rewards improve training of language models compared to the standard group‑relative policy optimization (GRPO) algorithm. Using a matched‑compute protocol that equalizes generated tokens and GPU‑hours across interventions, the authors find that dense rewards—such as Monte‑Carlo step‑wise scoring—do not outperform GRPO, and in some cases perform worse even when matched for compute. Instead, a simple strategy of training on prompts predicted to be near the 50% success threshold yields consistent accuracy gains across multiple models and settings.

By Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri, Sreehari Sankar, Wang Yang, Chaoda Song, Alexander Nemecek, Osama Zafar, Yuting Shao, Kai Ye, Biyao Zhang, Xinpeng Li, Shouren Wang, Erman Ayday, Jing Ma, Sanmukh Rao Kuppannagari, Vipin Chaudhary
arXiv AI
Jul 15

A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

arXiv:2607. 12640v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization (GRPO) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent.

By Chengguang Gan, Zhixi Cai, Yunhao Liang, Hanjun Wei, Shiwen Ni, Qinghao Zhang
arXiv AI
Oct 12

Unsupervised Multimodal Reward Modeling

The paper presents an unsupervised multimodal reward model built on a frozen Qwen3‑VL backbone. By adding a linear head to an intermediate layer and training it to rank a web caption higher than a perturbed version of itself, the authors avoid human preference labels and achieve competitive performance—outperforming GPT‑4o on several benchmarks with far fewer parameters and compute. The approach demonstrates that a carefully chosen intermediate layer, simple perturbation-based training, and verifier‑driven fine‑tuning can yield effective vision‑language rewards without costly annotation.

By Yueying Li, Jingxuan Fan, Kiant\'e Brantley, Sham M. Kakade, Hanlin Zhang