arXiv:2608.30724v1 Announce Type: cross Abstract: LLM agents are increasingly used to run autonomous ML experiments, iterating on target metrics with little human oversight. Prior work has documented...
arXiv:2609.08149v1 Announce Type: new Abstract: SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analys...
arXiv:2606. 07682v1 Announce Type: cross Abstract: AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments.
arXiv:2604. 23488v2 Announce Type: replace Abstract: Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the intended task, poses a critical challenge for Reinforcement Learning (RL) and the deployment of reasoning models.
arXiv:2604. 23488v3 Announce Type: replace Abstract: Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the intended task, poses a critical challenge for Reinforcement Learning (RL) and the deployment of reasoning models.
arXiv:2608.22103v1 Announce Type: new Abstract: As agents grow more capable and autonomous, their tendency to reward hack, satisfying a task's checks while violating its intent, becomes an increasing...
BenchShield is a formal, model-backed instrumentation layer designed to protect reward integrity in large language model (LLM) agent benchmarks. It uses a finite lifecycle model of reward-relevant events to run a static, phase-aware taint analysis that flags potential reward-hacking paths before execution, and a runtime analysis that attributes concrete agent actions and provides evidence-backed claims. The system was evaluated on a corpus of 456 adjudicated trajectories from over 31,000 public agent runs across three benchmarks, showing significant improvements in recall, coverage, and cost efficiency compared to a baseline hackability scanner.
arXiv:2606. 26300v1 Announce Type: new Abstract: A classical intuition holds that verifying a solution is easier than producing one.
arXiv:2604. 01527v4 Announce Type: replace-cross Abstract: Production deployment of AI coding agents requires fast, reproducible evaluation signals.
arXiv:2609.21562v1 Announce Type: cross Abstract: Coding agents can modify and test code across large software projects. Game development is a domain where agents must implement gameplay rules. A gam...
arXiv:2602. 20213v2 Announce Type: replace-cross Abstract: The evaluation of Large Language Models (LLMs) for code generation relies heavily on the quality and robustness of test cases.
arXiv:2607. 05188v1 Announce Type: new Abstract: A coding agent solving a software-engineering task spends dozens of steps reasoning, editing code, and running tests, yet little is known about what the underlying language model internally represents about the program it is working on.