The paper "Demystifying Reinforcement Learning Post-Training of Language Models" investigates how reinforcement learning (RL) post‑training enhances large language models (LLMs) for tasks such as reasoning, math, and coding. By isolating RL components in a controlled setting, the authors analyze how the base model’s prior distribution, reward granularity, prompt diversity, and model scale influence outcomes, using policy entropy to compare pre‑training, supervised fine‑tuning (SFT), and RL stages. The study clarifies the role of spurious rewards, the importance of the base model’s probability mass on desired behaviors, and how these factors interact to determine post‑training success, offering a practical primer for NLP researchers.
"whyItMatters":"The work provides a clearer understanding of RL post‑training mechanics, helping researchers and practitioners effectively apply RL to improve LLM capabilities."
By Donovan Clay, Saket Gollapudi, Sankar Harilal, Min Jang, Jacob Morrison, Sewoong Oh, Natasha Jaques
The paper reviews data strategies for large language models, emphasizing that data strategy—selecting, weighting, and scheduling training data—determines model capabilities within a fixed data budget. It treats high‑quality data as having training‑dependent utility, varying with the target objective, model, and training state. The authors organize existing methods by decision granularity, model dependence, and temporal adaptivity, comparing their computational costs and empirical support, and note limited evidence for transfer across model scales and end‑to‑end cost parity.
By Yuanjian Xu, Jianing Hao, Tianze Sun, Yongyao Duan, Sam Chen, Jinyi Han, Wanbo Zhang, Yuxuan Sun, Guang Zhang, Zhong Li
The paper introduces BehaviorTrace, an evaluation framework for assessing training‑data attribution in online reinforcement‑learning fine‑tuning of language models. Using a planted behavior on Qwen2.5‑1.5B, the authors show that many attribution signals are confounded by factors such as gradient magnitude and model fluency, and that only the gradient of trigger tokens consistently aligns with the true behavior across seeds. They provide a checklist for evaluating attribution methods and test existing estimators without proposing a new one.
By Amit Nautiyal
arXiv:2510. 10541v2 Announce Type: replace-cross Abstract: Current benchmarks are inadequate for evaluating progress in reinforcement learning (RL) for large language models (LLMs).
By Zihan Chen, Yiming Zhang, Hengguang Zhou, Zenghui Ding, Yining Sun, Cho-Jui Hsieh
arXiv:2601. 15353v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has achieved remarkable success in real-world decision-making across diverse domains, including gaming, robotics, online advertising, public health, and natural language processing.
By Asim H. Gazi, Yongyi Guo, Daiqi Gao, Ziping Xu, Kelly W. Zhang, Susan A. Murphy
Agnostics is a language‑agnostic post‑training pipeline that uses reinforcement learning with verifiable rewards (RLVR) to improve large language models on low‑resource programming languages. By rewriting unit‑test datasets into a language‑independent I/O format, providing a short configuration for compiling and running code, and employing a single verifier that judges code by observable behavior, Agnostics eliminates the need for language‑specific engineering. Applied to Lua, Julia, R, OCaml, and Fortran, it boosts Qwen‑3 4B to rival larger models, scales to diverse families, and achieves new state‑of‑the‑art pass@1 on MultiPL‑E and a new multi‑language LiveCodeBench.
By Aleksander Boruch-Gruszecki, Yangtian Zi, Zixuan Wu, Tejas Oberoi, Carolyn Jane Anderson, Joydeep Biswas, Arjun Guha
arXiv:2601. 12186v3 Announce Type: replace-cross Abstract: Multi-domain thinking verifiers trained via Reinforcement Learning with Verifiable Rewards (RLVR) are a cornerstone of modern post-training.
By Vatsal Venkatkrishna, Indraneil Paul, Iryna Gurevych
RH-Detect is a unified benchmark for detecting reward hacking in language models, aggregating 92,761 examples from eleven public datasets into a common schema with six behavior categories. It evaluates six off‑the‑shelf models on 5,021 open‑ended evaluation units, achieving a top AUROC of 0.962 and over 93% accuracy, though performance drops notably on multi‑turn tool‑use datasets. The study also shows that input format changes and single‑token fine‑tuning can influence detector performance, revealing gaps in deployment‑time monitoring.
By Junwei Quan, Evgenii Opryshko, Rohan Subramani, Igor Gilitschenski
SWE-Critix is a post‑training pipeline that creates generalist verifiers for coding‑agent trajectories. It uses supervised fine‑tuning followed by reinforcement learning to train 30B models on CodeForge‑Preview data, enabling the verifier to predict completion labels and generate natural‑language explanations. The approach raises classification accuracy from 57% to 78% on a held‑out test set.
By Jiajun Hu, Chun Yong Chong
The study examines how the rapid turnover of large language models (LLMs) undermines the effectiveness of AI‑detectors used to screen scientific manuscripts. By comparing 4,000 PNAS abstracts rewritten by 23 different LLM versions released between June 2023 and August 2026, the authors show that detectors trained on past versions can fail dramatically at generation boundaries, missing up to 96% of rewrites just after a sharp transition. Even detectors updated to newer versions can overlook rewrites of older models, and a commercial detector missed most rewrites of the newest version while flagging few human‑written abstracts.
By Kazuki Nakajima, Takayuki Mizuno
GrowMTP is a method that trains a draft head entirely within the reinforcement learning (RL) loop, using supervision from the RL verification step and a rollout distribution that is narrower than pretraining. By detaching draft‑head updates from the policy backbone, it enables online training of the draft head from scratch. Experiments on Qwen3‑4B, MiMo‑7B‑SFT, and Qwen3.5‑4B‑Base show rollout speedups ranging from 1.36× to 2.13× and overall end‑to‑end speedups from 1.20× to 1.60×, making it a modular acceleration component for RL frameworks lacking pretrained draft heads.
By Minghua He, Lingzhe Zhang, Yuan Liu, Xiao Zhou, Aiwei Liu
arXiv:2606. 14199v1 Announce Type: cross Abstract: Large language models are increasingly deployed as human simulators for interactive evaluation and social simulation.
By Xuhui Zhou, Weiwei Sun, Weihua Du, Jiarui Liu, Haojia Sun, Qianou Ma, Tongshuang Wu, Yiming Yang, Maarten Sap