The paper presents an unsupervised multimodal reward model built on a frozen Qwen3‑VL backbone. By adding a linear head to an intermediate layer and training it to rank a web caption higher than a perturbed version of itself, the authors avoid human preference labels and achieve competitive performance—outperforming GPT‑4o on several benchmarks with far fewer parameters and compute. The approach demonstrates that a carefully chosen intermediate layer, simple perturbation-based training, and verifier‑driven fine‑tuning can yield effective vision‑language rewards without costly annotation.
By Yueying Li, Jingxuan Fan, Kiant\'e Brantley, Sham M. Kakade, Hanlin Zhang
Poolwise is a protocol that evaluates reward models for selecting training data across different prompts by choosing a fixed fraction of responses based on score and measuring the quality of the selected set with external labels. It reports within‑prompt ranking separately and shows that training each prompt’s mean score to match its average response rating improves selection over permuted or omitted mean targets in all tested seeds at 30% retention. In grouped HelpSteer experiments, the model with the highest within‑prompt accuracy selects a lower‑rated set than the correct‑mean model, and subtracting prompt means from reference responses preserves local rankings but reduces selected helpfulness for several reward models and judges.
whyItMatters":"Poolwise offers a direct method to compare reward models and score transformations for data selection, addressing the limitation of existing benchmarks that only compare responses within the same prompt."
By Mateusz Modrzejewski, Tomasz Radzikowski
arXiv:2607. 16195v1 Announce Type: new Abstract: We identify a structured confound in Reinforcement Learning from Human Feedback (RLHF).
By Elena Kopteva, Vitaliy Hlynianyi-Zhuk
arXiv:2606. 18963v1 Announce Type: new Abstract: We study online reward-punishment learning when the environment provides no scalar reward or evaluative label.
By Zirong Li
arXiv:2608.30005v1 Announce Type: new
Abstract: Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific c...
By Fengyu Xie, Yilun Zhao, Bingsen Chen, Arman Cohan, Chen Zhao
arXiv:2606. 03238v1 Announce Type: cross Abstract: Reinforcement learning from human feedback (RLHF) makes large-scale post-training possible by replacing an underspecified human objective with learned and scalable proxies.
By Zelalem Abahana
arXiv:2607. 18163v1 Announce Type: cross Abstract: PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative.
By Chinmay Rane, Kanishka Tyagi, Michael Manry
arXiv:2608. 11669v1 Announce Type: cross Abstract: Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer.
By Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Ying Liu
arXiv:2608. 13564v1 Announce Type: new Abstract: Evaluating language-model agents at scale increasingly relies on a second language model as an automatic judge, because the gold signal, an executable environment reward, is expensive, slow, or unavailable at deployment time.
By Darragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan
ReCAST is a method for assigning credit to multiple rewards during diffusion model training by using a reward-by-timestep weight matrix that respects user-specified reward budgets while ensuring equal total weight per denoising step. It allocates weight based on each reward’s informativeness, measured by its Rényi discriminability gain at each step, allowing rewards to contribute more when they are most informative. Experiments on SD3.5‑Medium with two four‑reward settings show that ReCAST improves or matches training rewards, enhances held‑out judges, and is preferred by an independent LLM‑as‑a‑Judge, indicating generalizable benefits.
By Yihang Chen, Yuanhao Ban, Kuei-Chun Kao, Cho-Jui Hsieh
arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.
By Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao
Terminal‑RSI is a new framework for terminal agents that alternates Recursive Self‑Rewrite (RSR) of verified trajectories with Recursive Policy‑Refine (RPR) that inherits a critic. It introduces TB‑GAE for token‑specific baselines, Family Dense Reward (FDR) and Success‑Prioritized Reward (SPR) to calibrate progress, and uses a warm‑up critic, recursive handoff, and stage‑specific learning rates to coordinate the two refinement phases. The five‑stage pipeline applied to Qwen3.8‑27B produces Terminal‑RSI‑27B, achieving 19.84 % on Terminal‑Bench 4.0, 86.52 % on Terminal‑Bench 2.1, 16.95 % on Terminal‑Bench 3.0, and 79.00 % on Terminal‑Bench Hard, while improving process reward and strict completion on LHTB.
By Junyao Yang, Zongxia Li, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Ruiwen Zhou, Haitao Mi, Shuicheng Yan, Leowei Liang