arXiv Computation and Language By Guowei Xu, Mert Yuksekgonul, James Zou

Sparse Reward Subsystem in Large Language Models

Read the original on arXiv Computation and Language →

The paper demonstrates that reward-related information in large language models is concentrated in a sparse subset of neurons. Two types of neurons are identified: value neurons, which predict state value, and dopamine neurons, which encode step-level temporal difference errors. These neurons form a reward subsystem that is robust across datasets and models, and can be used to predict model confidence and guide inference-time search.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
Jun 2

Latent Reward Steering: An Adaptive Inference-Time Framework that Implicitly Promotes Cognitive Behaviors in Reasoning LLMs

arXiv:2606. 00726v1 Announce Type: new Abstract: Strong reasoning depends not only on model knowledge but also on how effectively cognitive behaviors are deployed during generation.

By Jiakang Li, Guanyu Zhu, Can Jin, Chenxi Huang, Dexu Yu, Ronghao Chen, Yang Zhou, Hongwu Peng, Xuanqi Lan, Dimitris N. Metaxas, Youhua Li
arXiv AI
Oct 2

Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States

The paper introduces POISE, a reinforcement learning algorithm that uses a model’s internal states as a value estimator to reduce variance in reinforcement learning with verifiable rewards (RLVR). By employing a lightweight probe that reads internal signals during the forward pass, POISE predicts baselines online and uses a cross‑rollout construction to keep gradients unbiased. Experiments on Qwen3‑4B and OLMo3‑7B‑Instruct‑DPO across six domains show POISE outperforms existing RLVR baselines, offering more stable training and a value model that generalizes across tasks and scales with the policy.

By Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh, Jeonghoon Shim, Yohan Jo
arXiv AI
Oct 16

Progress Reward Model for Reinforcement Learning via Large Language Models

The paper introduces PRM4RL, a framework that combines task planning and dense reward generation using large language models for reinforcement learning. It decomposes complex tasks into manageable subtasks and employs a fine‑grained progress function to monitor execution, which is then used to construct a Progress Reward Model for potential‑based reward shaping. Experiments on robotics control tasks show that PRM4RL outperforms existing LLM‑based planning and reward methods, achieving state‑of‑the‑art performance.

By Xiuhui Zhang, Ning Gao, Xingyu Jiang, Yihui Chen, Yuheng Pan, Mohan Zhang, Yue Deng