arXiv Machine Learning

The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works

arXiv:2607. 21273v2 Announce Type: replace Abstract: Dense per-step supervision is the standard remedy for sparse-reward long-horizon LLM agents: reward the policy for predicting its next observation, which looks provably safe under potential-based shaping.

arXiv Machine Learning

Jul 24

The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works

arXiv:2607. 21273v1 Announce Type: new Abstract: Dense per-step supervision is an appealing remedy for sparse-reward, long-horizon LLM agents: reward the agent for predicting its next observation, and memory should follow.

By Yu Wang

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 18

Online Reward-Punishment Learning from Fixed-Channel Perceptual Event Streams without Environment Rewards

arXiv:2606. 18963v1 Announce Type: new Abstract: We study online reward-punishment learning when the environment provides no scalar reward or evaluative label.

By Zirong Li

agents

arXiv Machine Learning

Jul 8

More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges

arXiv:2607. 05904v1 Announce Type: new Abstract: Training a language model against its own reference-free judgments (the premise of self-rewarding, self-play, and LLM-as-a-judge pipelines) assumes a model's verdict on a shown answer tracks correctness.

By Chenyu Zhou

llms reinforcement-learning

arXiv Machine Learning

Jun 25

BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents

arXiv:2606. 25556v1 Announce Type: cross Abstract: Stepwise group-based RL is an attractive way to train long-horizon LLM agents without a learned critic: it reuses multiple sampled rollouts to estimate local advantages.

By Hanyang Wang, Weijieying Ren, Yuxiang Zhang, Ding Cao, Zhizhao Zeng, Ke Zeng, Tianxiang Zhao

llms agents reinforcement-learning

arXiv Machine Learning

Aug 4

Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning

arXiv:2608. 00301v1 Announce Type: new Abstract: Error-penalized scoring rules ($+1$ for a correct answer, $-\lambda$ for a wrong one, $0$ for abstaining) are increasingly prescribed against hallucination: a rational agent facing such a rule answers exactly when its correctness probability exceeds Chow's threshold $t^\ast=\lambda/(1+\lambda)$.

By Xujun Che, Yuchen Yuan, Weida Zhao, Chenyang Yu

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jul 15

Adaptive Compute in Latent World Models: When Depth Helps, Hurts, or Doesn't Matter

arXiv:2607. 10203v2 Announce Type: replace-cross Abstract: Adaptive-compute world models -- early-exit or mixture-of-depths predictors that spend variable depth per step -- assume depth buys better predictions and can be routed adaptively.

By Achyuthan Sivasankar

benchmarks

arXiv Machine Learning

Aug 12

Detecting an Effect Is Not Learning to Act on It: A Reward-SNR Floor for LLM Acquisition Agents

arXiv:2608. 10441v1 Announce Type: new Abstract: Many pipelines can pay a per-example cost to acquire an auxiliary, model-derived observation -- an LLM's structured reasoning, a slow oracle, an expensive measurement -- and then must decide when the acquired signal is worth using.

By Ying Yuan

llms rag agents

arXiv AI

Jul 21

OR Else: A Differentiable Trust Region for Policy Optimization

arXiv:2607. 18163v1 Announce Type: cross Abstract: PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative.

By Chinmay Rane, Kanishka Tyagi, Michael Manry

llms reinforcement-learning

arXiv AI

Jul 14

When Does Depth Survive Composition? Compute--Quality Regimes in Latent World Models

arXiv:2607. 10203v1 Announce Type: cross Abstract: Adaptive-compute world models -- early-exit or mixture-of-depths predictors that spend variable depth per step -- assume depth buys better predictions and can be routed adaptively.

By Achyuthan Sivasankar

benchmarks

arXiv Machine Learning

Jul 14

When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR

arXiv:2607. 11022v1 Announce Type: new Abstract: The test suites used as RLVR rewards for code have natural false positives: per-task, persistent, asymmetric errors that accept the same wrong programs every time they appear, unlike the symmetric or resampled noise assumed by existing noise-robustness analyses.

By Chuyifei Zhang

arXiv AI

Jul 15

A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

arXiv:2607. 12640v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization (GRPO) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent.

By Chengguang Gan, Zhixi Cai, Yunhao Liang, Hanjun Wei, Shiwen Ni, Qinghao Zhang

llms rag agents reinforcement-learning multimodal

arXiv Machine Learning

Jul 15

When Does Reward Teach State? A Hidden-Automaton Instrument and the Group-Language Boundary

arXiv:2607. 11953v1 Announce Type: new Abstract: Does a reinforcement-learning agent that earns high reward represent its task's latent state, or only a reward-correlated shortcut?

By Jim Allchin

agents reinforcement-learning

Related stories