arXiv AI
3d ago

Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models

The paper proposes a two‑channel constrained GRPO framework that treats safety as explicit budgets on harmful assistance and over‑refusal rates, using Lagrange multipliers and a monotone mapping of scorer outputs. It demonstrates that this calibration reduces harmful responses from 8.9% to 4.8% on a Qwen3‑VL‑2B policy while keeping over‑refusal low, outperforming standard GRPO and uncalibrated budgets across multiple benchmarks.

By Bo Han
arXiv AI
Oct 12

Unsupervised Multimodal Reward Modeling

The paper presents an unsupervised multimodal reward model built on a frozen Qwen3‑VL backbone. By adding a linear head to an intermediate layer and training it to rank a web caption higher than a perturbed version of itself, the authors avoid human preference labels and achieve competitive performance—outperforming GPT‑4o on several benchmarks with far fewer parameters and compute. The approach demonstrates that a carefully chosen intermediate layer, simple perturbation-based training, and verifier‑driven fine‑tuning can yield effective vision‑language rewards without costly annotation.

By Yueying Li, Jingxuan Fan, Kiant\'e Brantley, Sham M. Kakade, Hanlin Zhang
arXiv AI
3d ago

CARE-PO: Controllable Tool-Call Efficiency for LLM Agents

The paper introduces CARE-PO, a method for controlling tool‑call efficiency in large language model agents. By anchoring an efficiency reward to a fixed reference budget tied to task difficulty and injecting successful trajectories into training rollouts, CARE‑PO prevents the collapse to zero tool calls and allows a single training run to produce multiple operating points balancing efficiency and capability. Experiments on the GAIA benchmark and other multi‑hop tasks show that CARE‑PO improves both success rates and call counts across several model sizes.

By Xiaoyang Ye, Xiao Yao, Hairui Zhu, Yiying Yang, Bei Gan, Jinliang Lu, Tengjin Weng, Yuan Yao, Wenhao Jiang
arXiv Machine Learning
Oct 3

Same Reward, Different Skills: When Multimodal RL Learns to Look

The paper demonstrates that reinforcement learning with verifiable rewards (RLVR) can improve vision‑language benchmark performance even when models are trained without visual input. When real images are introduced at test time, models trained blind recover about half of the performance gain at 3B parameters and nearly four‑fifths at 7B, but extended real‑image training can erode grounding while benchmark gains persist. The authors propose a visual resolvability rule and show that requiring visual evidence for correct answers leads to significant improvements in target discovery and generalization to unseen question types, while controls confirm that the gains stem from actual visual grounding rather than artifacts.

By Haocun Ye, Xinlong Jiang, Qile Chen, Bingyu Wang, Teng Zhang, Shubai Chen, Tingyu Wu, Zhenkun Zheng, Yiqiang Chen