arXiv Machine Learning September 10, 2026 By Esther Xin

Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rollouts

Read the original on arXiv Machine Learning →

The paper investigates whether verifier errors are independent within groups of completions generated by the Qwen2.5-1.5B model on benchmark datasets. Analyses of 24,998 groups of eight completions reveal a pooled within‑group verifier‑error correlation of 0.530, indicating significant clustering of errors. The degree of dependence varies by answer form, with fractions, radicals, symbolic expressions, and intervals showing stronger clustering than unit annotations and percent signs, and up to 0.83% of groups exhibit disagreement in advantage signs across rule‑based verifier configurations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

reinforcement-learning

arXiv Machine Learning

Sep 2

Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR

arXiv:2609.01354v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free text ans...

By Esther Xin

reinforcement-learning benchmarks

arXiv Computation and Language

6d ago

Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization

arXiv:2610.34951v1 Announce Type: cross Abstract: Recent methods for Reinforcement Learning with Verifiable Rewards (RLVR) use rollout diversity to improve reasoning performance or training efficienc...

By Sohyun Son, Yerim Hong, Jaeyun Park, Mujeen Sung

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 5

A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR

arXiv:2606. 05932v1 Announce Type: cross Abstract: Reinforcement learning from verifiable rewards (RLVR) improves reasoning even when the reward signal is spurious -- assigning credit to the group-plurality answer rather than a ground-truth verifier.

By Yuze Gao

reinforcement-learning safety

arXiv Machine Learning

Sep 30

Learned Reporting Preferences in RLVR Can Conflict with the Current Request

arXiv:2609.36587v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a prominent approach for improving language-model performance on reasoning tasks using...

By Yupeng Chang, Wenxuan Zhang, Yuan Wu

reinforcement-learning

arXiv Machine Learning

Jul 7

Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models

arXiv:2607. 02869v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for improving mathematical reasoning in language models.

By Anagha Radhakrishna Palandye, Rebecca Glick, Osheen Kaul

llms reinforcement-learning fine-tuning

arXiv AI

Aug 20

Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning

The paper introduces Group‑Calibrated On‑Policy Distillation (GC‑OPD), a method that aligns token‑level teacher guidance with trajectory‑level verifier rewards for long‑context reasoning tasks. GC‑OPD normalizes rewards within rollout groups, uses the signed teacher‑verifier disagreement as a residual, and distributes this residual across tokens via Relative‑Advantage‑Based Credit Assignment (RACA). Experiments on five long‑context benchmarks show that GC‑OPD improves Qwen3‑4B and Qwen3‑8B checkpoints from 29.08/35.12 to 40.47/44.65, outperforming vanilla OPD and demonstrating the effectiveness of group‑relative residual calibration.

By Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao, Yuezhi Zhou

efficiency benchmarks

One email a morning, machine-written

Related stories

Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR

Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization

A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR

Learned Reporting Preferences in RLVR Can Conflict with the Current Request

Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models

Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning