arXiv AI June 30, 2026 By Subramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary

Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models

arXiv:2606. 30627v1 Announce Type: cross Abstract: Conservative offline training is widely advocated as a safe foundation for subsequent online adaptation: if a policy stays close to well-supported behaviour, the argument goes, it is less likely to exploit imperfections in a learned reward model.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jun 3

When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

arXiv:2606. 03238v1 Announce Type: cross Abstract: Reinforcement learning from human feedback (RLHF) makes large-scale post-training possible by replacing an underspecified human objective with learned and scalable proxies.

By Zelalem Abahana

llms reinforcement-learning safety

arXiv AI

6d ago

Post-Training with Policy Gradients: Optimality and the Base Model Barrier

arXiv:2603. 06957v2 Announce Type: replace-cross Abstract: We study post-training linear autoregressive models with outcome and process rewards.

By Alireza Mousavi-Hosseini, Murat A. Erdogdu

reinforcement-learning

arXiv AI

1d ago

Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization

arXiv:2608. 15445v1 Announce Type: new Abstract: When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization.

By Suyash Maniyar, Armaan Sandhu, Abhishek Mishra

llms benchmarks

arXiv AI

Jul 14

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

arXiv:2607. 10139v1 Announce Type: cross Abstract: Selecting the correct answer from a pool of candidate reasoning chains is the engine of test-time scaling, yet the standard selectors each carry a cost: self-consistency inherits the errors of the single model it resamples, and trained reward models need labeled data and transfer poorly off-distribution.

By Ning Liu

llms benchmarks

arXiv AI

Jul 22

Measuring Reward-Seeking via Contrastive Belief Updates

arXiv:2607. 18966v1 Announce Type: new Abstract: Language models trained with reinforcement learning may learn to optimize the grader's judgment rather than the intended objective.

By Axel H{\o}jmark, J\'er\'emy Scheurer, Evgenia Nitishinskaya, Felix Hofst\"atter, Jason Wolfe, Theodore Ehrenborg, Bronson Schoen, Alexander Meinke

llms reinforcement-learning fine-tuning safety

arXiv AI

6d ago

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

arXiv:2608. 11669v1 Announce Type: cross Abstract: Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer.

By Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Ying Liu

llms reinforcement-learning benchmarks safety

Related stories

When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

Post-Training with Policy Gradients: Optimality and the Base Model Barrier

Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

Measuring Reward-Seeking via Contrastive Belief Updates

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL