The paper introduces Test‑Time Policy Optimization (TTPO), an approach that enables large language models to improve mathematical reasoning without relying on ground‑truth labels. TTPO uses majority‑vote pseudo‑labels and an asymmetric objective: it distills rollouts that agree with the pseudo‑label via On‑Policy Self‑Distillation and penalizes disagreeing rollouts with Grouped Reinforcement Learning. Token‑level selection further refines the process, down‑weighting already‑converged positions during distillation and penalizing only confident errors during RL. Experiments show that TTPO matches label‑supervised OPSD on five competition‑level benchmarks, boosts Qwen3‑1.7B from 38.0 % to 45.2 % in test‑time training, and achieves significant gains without explicit reasoning steps, while also generalizing well across tasks.
By Aozhe Wang, Zhengxi Lu, Jianze Wang, Shangke Lv, Ying Liu, Weiming Lu, Jun Xiao, Yueting Zhuang, Hua Yang, Qianglong Chen, Yongliang Shen
The paper introduces Test‑Time Policy Optimization (TTPO), a method that enables large language models to improve mathematical reasoning during inference without relying on ground‑truth labels. TTPO uses majority‑vote pseudo‑labels to guide an asymmetric objective: agreeing rollouts are distilled via On‑Policy Self‑Distillation, while disagreeing rollouts are penalized with Grouped Reinforcement Learning, with token‑level selection refining both branches. Experiments show that TTPO matches label‑supervised OPSD on five benchmarks, boosts Qwen3‑1.7B from 38.0 % to 45.2 % in test‑time training, and achieves significant gains without explicit reasoning steps, demonstrating strong cross‑task generalization.
The paper introduces probe-driven test-time reinforcement learning (TTRL) for code generation, using probe inputs derived from problem statements to evaluate candidate programs and define a Probe Consensus Reward (PCR). To address PCR’s unreliability and prevent reward hacking, the authors propose Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which applies rank masking and an entropy ceiling to produce conservative policy updates. Experiments on coding benchmarks show that ERPO significantly improves pass@1 and pass@k metrics in both in-domain adaptation and zero-shot transfer scenarios.
By Jiacheng Xu, Feng Chen, Xiuneng Xu, Bo An
arXiv:2608. 10209v1 Announce Type: new Abstract: Feedback signals used to train Large Language Models (LLMs) are the primary driver of their behavior and our main lever for instilling alignment with human values and objectives.
By Alec Harris, Kasey Corra, Archie Chaudhury, Yixiong Hao
The paper introduces T$^3$RL, a tool‑verification framework for test‑time reinforcement learning (TTRL) that mitigates the false‑popular failure mode by using external tool evidence to upweight verified rollouts during voting. By grounding pseudo‑label construction in verified evidence, T$^3$RL produces more reliable pseudo‑labels and improves performance over standard TTRL on math benchmarks such as MATH‑500, AMC, and AIME 2024. The approach positions T$^3$RL as a verified online data synthesizer, highlighting the importance of tool verification for reliable online adaptation and demonstrating extensibility to other verifiable domains.
By Ruotong Liao, Nikolai R\"ohrich, Xiaohan Wang, Yuhui Zhang, Yasaman Samadzadeh, Volker Tresp, Serena Yeung-Levy
arXiv:2607. 28457v1 Announce Type: cross Abstract: Scaling test-time computation can improve language-model reasoning, but uniform budgets waste computation on easy inputs, while verifier-guided refinement relies on external feedback.
By Hongyu Chen, Liang Lin, Guangrun Wang
arXiv:2608. 03119v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but typically relies on ground-truth (GT) answers, limiting scalability.
By Yongshi Ye, Liang Zhang, Yidong Chen, Xiaodong Shi, Biao Fu
arXiv:2606. 27369v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) for training LLMs typically rely on ground-truth answers to assign rewards, limiting their applicability to tasks where the ground-truth solution is unknown.
By Yingyu Lin, Qiyue Gao, Nikki Lijing Kuang, Xunpeng Huang, Kun Zhou, Tongtong Liang, Zhewei Yao, Yi-An Ma, Yuxiong He
arXiv:2610.33512v1 Announce Type: new
Abstract: Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighte...
By Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose, Eunho Yang
arXiv:2602. 18037v2 Announce Type: replace-cross Abstract: Reinforcement Learning from Human Feedback (RLHF) or Verifiable Rewards (RLVR) are two key steps in the post-training of modern Language Models (LMs).
By Johannes Ackermann, Michael Noukhovitch, Takashi Ishida, Masashi Sugiyama
arXiv:2605. 02909v2 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs).
By Kazuki Egashira, Mark Vero, Jasper Dekoninck, Florian E. Dorner, Robin Staab, Martin Vechev
arXiv:2609.16660v1 Announce Type: new
Abstract: Test-time reinforcement learning adapts a model on its own unlabeled test set using majority-vote pseudo-labels and has shown strong results in mathema...
By Kailong Fan, Anqi Pu, Yichen Wu, Wanhua Li, Yicong Li, Hanspeter Pfister, Huafeng Liu, Xiang Li, Quanzheng Li, Ning Guo