arXiv AI
CheatBench: Measuring Reward Gaming in AI Agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 1
BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
arXiv:2608.30724v1 Announce Type: cross Abstract: LLM agents are increasingly used to run autonomous ML experiments, iterating on target metrics with little human oversight. Prior work has documented...
arXiv AI
Jun 8
Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests
arXiv:2606. 07379v1 Announce Type: cross Abstract: A growing failure mode in agent evaluation and training is that models can achieve high evaluation scores by exploiting shortcuts instead of solving the intended task, producing deceptive performance.
arXiv AI
Sep 12
BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
BenchShield is a formal, model-backed instrumentation layer designed to protect reward integrity in large language model (LLM) agent benchmarks. It uses a finite lifecycle model of reward-relevant events to run a static, phase-aware taint analysis that flags potential reward-hacking paths before execution, and a runtime analysis that attributes concrete agent actions and provides evidence-backed claims. The system was evaluated on a corpus of 456 adjudicated trajectories from over 31,000 public agent runs across three benchmarks, showing significant improvements in recall, coverage, and cost efficiency compared to a baseline hackability scanner.
arXiv AI
Jun 4
Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
arXiv:2606. 04923v1 Announce Type: cross Abstract: Rubric-based reinforcement learning (RL) uses an LLM-as-a-Judge (LaaJ) to score model outputs according to rubrics as rewards.
arXiv AI
Jun 16
Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
arXiv:2606. 15385v1 Announce Type: new Abstract: Reward hacking, where AI systems exploit misspecified objectives to achieve high reward without satisfying intended goals, remains a central challenge in AI safety.
arXiv Computation and Language
Sep 10
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
arXiv:2605.21384v2 Announce Type: replace-cross Abstract: As long-horizon coding agents produce more code than any developer can review, oversight collapses onto a single surface: the automated test...