arXiv Machine Learning By Mingxiang Shen, Jinhao Li, Jiaming Xu, Yibo Han, Guohao Dai, Yijia Zhang, Ningyi Xu

TRUER: Test-Time Reward Correction Using an Error-Aware Rectifier for Reinforcement Learning

Read the original on arXiv Machine Learning →

TRUER is a framework that corrects test‑time rewards in reinforcement learning by training a reward rectifier with limited supervision. The rectifier learns to distinguish correctness from consensus, focusing on cases where majority voting is wrong. After training, the rectifier is frozen and used to adjust rewards during test‑time adaptation, consistently outperforming naive TTRL across all evaluated settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language
Aug 28

TTPO: Test-Time Policy Optimization

The paper introduces Test‑Time Policy Optimization (TTPO), an approach that enables large language models to improve mathematical reasoning without relying on ground‑truth labels. TTPO uses majority‑vote pseudo‑labels and an asymmetric objective: it distills rollouts that agree with the pseudo‑label via On‑Policy Self‑Distillation and penalizes disagreeing rollouts with Grouped Reinforcement Learning. Token‑level selection further refines the process, down‑weighting already‑converged positions during distillation and penalizing only confident errors during RL. Experiments show that TTPO matches label‑supervised OPSD on five competition‑level benchmarks, boosts Qwen3‑1.7B from 38.0 % to 45.2 % in test‑time training, and achieves significant gains without explicit reasoning steps, while also generalizing well across tasks.

By Aozhe Wang, Zhengxi Lu, Jianze Wang, Shangke Lv, Ying Liu, Weiming Lu, Jun Xiao, Yueting Zhuang, Hua Yang, Qianglong Chen, Yongliang Shen
Hugging Face Trending Papers
Aug 27

TTPO: Test-Time Policy Optimization

The paper introduces Test‑Time Policy Optimization (TTPO), a method that enables large language models to improve mathematical reasoning during inference without relying on ground‑truth labels. TTPO uses majority‑vote pseudo‑labels to guide an asymmetric objective: agreeing rollouts are distilled via On‑Policy Self‑Distillation, while disagreeing rollouts are penalized with Grouped Reinforcement Learning, with token‑level selection refining both branches. Experiments show that TTPO matches label‑supervised OPSD on five benchmarks, boosts Qwen3‑1.7B from 38.0 % to 45.2 % in test‑time training, and achieves significant gains without explicit reasoning steps, demonstrating strong cross‑task generalization.

arXiv Machine Learning
Sep 10

Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation

The paper introduces probe-driven test-time reinforcement learning (TTRL) for code generation, using probe inputs derived from problem statements to evaluate candidate programs and define a Probe Consensus Reward (PCR). To address PCR’s unreliability and prevent reward hacking, the authors propose Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which applies rank masking and an entropy ceiling to produce conservative policy updates. Experiments on coding benchmarks show that ERPO significantly improves pass@1 and pass@k metrics in both in-domain adaptation and zero-shot transfer scenarios.

By Jiacheng Xu, Feng Chen, Xiuneng Xu, Bo An
arXiv Computation and Language
Aug 27

Tool Verification for Test-Time Reinforcement Learning

The paper introduces T$^3$RL, a tool‑verification framework for test‑time reinforcement learning (TTRL) that mitigates the false‑popular failure mode by using external tool evidence to upweight verified rollouts during voting. By grounding pseudo‑label construction in verified evidence, T$^3$RL produces more reliable pseudo‑labels and improves performance over standard TTRL on math benchmarks such as MATH‑500, AMC, and AIME 2024. The approach positions T$^3$RL as a verified online data synthesizer, highlighting the importance of tool verification for reliable online adaptation and demonstrating extensibility to other verifiable domains.

By Ruotong Liao, Nikolai R\"ohrich, Xiaohan Wang, Yuhui Zhang, Yasaman Samadzadeh, Volker Tresp, Serena Yeung-Levy