arXiv:2602. 18037v2 Announce Type: replace-cross Abstract: Reinforcement Learning from Human Feedback (RLHF) or Verifiable Rewards (RLVR) are two key steps in the post-training of modern Language Models (LMs).
The paper surveys 58 studies on verifier errors in reinforcement learning with verifiable rewards (RLVR), examining how such errors affect training and which mitigations are effective. It finds that the type of error—rather than its frequency—has a larger impact on learning dynamics, with random errors mainly slowing progress and false positives potentially causing learning plateaus or collapse. The authors compile findings into comparison tables, reconcile conflicting reports, and propose a reporting protocol to standardize future research.
arXiv:2608. 13564v1 Announce Type: new Abstract: Evaluating language-model agents at scale increasingly relies on a second language model as an automatic judge, because the gold signal, an executable environment reward, is expensive, slow, or unavailable at deployment time.
arXiv:2609.39533v1 Announce Type: new Abstract: During reinforcement learning with verifiable rewards (RLVR), large language models (LLMs) can exploit loopholes in their environments to obtain high r...
arXiv:2607. 02755v1 Announce Type: cross Abstract: Training AIs to be risk-averse in resources could offer a failsafe in the event that AIs turn out misaligned.
The paper investigates whether the reported out‑of‑domain performance gains of Reinforcement Learning with Verifiable Rewards (RLVR) are due to improved answer quality or simply to changes in answer formatting. By comparing scores from standard extractors with a custom extractor that distinguishes answers inside a oxed{} notation, the authors find that a large portion (about 82%) of the apparent gains on 7B SimpleRL models vanish, indicating that the gains are largely due to formatting rather than better answers. In contrast, a 32B model trained with a different recipe shows minimal format‑related gains, suggesting that the effect depends on the training setup.
arXiv:2607. 16195v1 Announce Type: new Abstract: We identify a structured confound in Reinforcement Learning from Human Feedback (RLHF).
The paper introduces Dynalize, an online penalty system that mitigates reward hacking in preference reward models during policy training. By having a language model periodically analyze rollouts and generate penalties for behaviors the reward model inadvertently rewards, Dynalize prevents the policy from inflating scores without improving quality. Experiments in clinical counseling and creative writing show that Dynalize sustains longer training and yields better downstream performance, suppressing undesirable patterns even in multi‑turn settings and after retraining the reward model.
arXiv:2610.19298v1 Announce Type: new Abstract: Reinforcement learning for code usually rewards passing tests, which a model can do while remaining wrong on the inputs the tests do not cover. A progr...
arXiv:2607. 18966v1 Announce Type: new Abstract: Language models trained with reinforcement learning may learn to optimize the grader's judgment rather than the intended objective.
arXiv:2607. 11022v1 Announce Type: new Abstract: The test suites used as RLVR rewards for code have natural false positives: per-task, persistent, asymmetric errors that accept the same wrong programs every time they appear, unlike the symmetric or resampled noise assumed by existing noise-robustness analyses.
arXiv:2609.01354v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free text ans...