The paper introduces Step-wise Rubrics as Rewards (SRaR), a reinforcement learning framework that assigns rubric-based scores to individual reasoning steps in large language models. By normalizing step scores and combining them with outcome rewards through a decoupled advantage estimator, SRaR preserves criterion-level structure and mitigates issues like rewarding flawed steps or encouraging endless self‑correction. Experiments on 16K problems and six math benchmarks show that SRaR outperforms prior rubric methods, improving accuracy, increasing faithful reasoning rates, and reducing self‑correction loops.
By Weichu Xie, Haozhe Zhao, Wenpu Liu, Yongfu Zhu, Liang Chen, Minghao Ye, Zirong Chen, Yuqi Xu, Shuai Dong, Ziyue Wang, Xinbo Xu, Kean Shi, Ruoyu Wu, Xiaoying Zhang, Wenqi Shao, Baobao Chang, Nan Duan, Jiaqi Wang
arXiv:2602. 20710v2 Announce Type: replace Abstract: Inspecting Chain-of-Thought reasoning is among the most common means of understanding why an LLM produced its output.
By Peter Hase, Christopher Potts
arXiv:2605. 03862v4 Announce Type: replace Abstract: Reinforcement learning with verifiable rewards has become a common way to improve explicit reasoning in large language models, but final-answer correctness alone does not reveal whether the reasoning trace is faithful, reliable, or useful to the model that consumes it.
By Tianyang Han, Hengyu Shi, Junjie Hu, Xu Yang, Zhiling Wang, Junhao Su
arXiv:2607. 11506v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards.
By Xiaojian Liu, Han Xu, Jianqiang Xia, Zhixuan Li, Ke Xu, Yiwei Dai, Xinran Chen, Changwo Wu, Yuchen Li
arXiv:2604.23351v2 Announce Type: replace-cross
Abstract: Whether intermediate reasoning is computationally useful or merely explanatory depends on whether chain-of-thought (CoT) tokens contain task-...
By Houman Mehrafarin, Amit Parekh, Ioannis Konstas
The paper investigates how reward misspecification in reinforcement learning harms the ability to monitor chain-of-thought (CoT) reasoning in language models. By creating a testbed with true and misspecified rewards, the authors show that models can hack the reward while still producing CoTs that appear aligned, leading to low monitorability. They propose a mitigation strategy that collects diverse reward rollouts, infers a reward description with a language model, and injects this description into the input, improving monitorability by up to 42% absolute across models and tasks.
By Tianyi Lorena Yan, John Hewitt
arXiv:2508. 02178v3 Announce Type: replace Abstract: Large reasoning models (LRMs) often exhibit overthinking, producing verbose Chain-of-Thought (CoT) traces that increase inference cost and obscure the underlying reasoning process.
By Taihang Zhen, Jialiang Hong, Kai Chen, Guang Yang, Junlan Feng, Wenpeng Zhu, Jing Huo, Yang Gao, Depeng Wang, Haitao Wan, Xi Yang, Fanyu Meng, Yuyao Zhang, Ji Qi, Xiangyu Zhou
Cliff is a reward‑shaping method for reinforcement learning with verifiable rewards that identifies the first mistake in a language model’s reasoning process using an off‑the‑shelf teacher. By splitting each rollout into a correct prefix and an incorrect suffix, Cliff assigns token‑level advantages—positive for correct tokens and negative for the rest—providing fine‑grained supervision. Across 12 scenarios, Cliff outperforms on‑policy distillation by 15% and standard GRPO by 7%, even when the teacher is only modestly capable.
By Peixuan Han, Runhui Wang, Ketan Ramaneti, Jie Hao, Gerald Friedland, Chris Kong
The paper introduces EquivAug, a data augmentation framework that applies symbolic equivalence transformations to chain-of-thought (CoT) solutions, generating diverse yet provably correct reasoning paths for supervised fine-tuning (SFT). By converting CoT solutions into formal symbolic representations, applying equivalence transformations, and reconstructing natural-language explanations, EquivAug produces training data that varies in reasoning steps rather than surface-level lexical differences. Experiments on GSM8K and FOLIO show that EquivAug consistently achieves the highest reasoning path diversity after SFT and the strongest accuracy after GRPO, establishing a positive correlation between CoT diversity during SFT and downstream reinforcement learning with verifiable rewards (RLVR) performance.
By Xiaoou Liu, Tiejin Chen, Hanyang Chen, Yuan Ling, Hua Wei
arXiv:2607. 16097v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it.
By Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky, Pavel Izmailov
arXiv:2603. 09803v2 Announce Type: replace Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally, potentially reinforcing flawed traces that arrive at correct answers by chance.
By Tiehua Mei, Minxuan Lv, Leiyu Pan, Zhenpeng Su, Hongru Hou, Hengrui Chen, Ao Xu, Deqing Yang
arXiv:2610.34951v1 Announce Type: cross
Abstract: Recent methods for Reinforcement Learning with Verifiable Rewards (RLVR) use rollout diversity to improve reasoning performance or training efficienc...
By Sohyun Son, Yerim Hong, Jaeyun Park, Mujeen Sung