The paper introduces AdaptRubric, a Coarse-to-Fine Rubrics Framework designed to create task‑adaptive judging criteria for GUI reward modeling. It first retrieves a category‑level coarse rubric by mapping instructions to a GUI task family, then generates an instance‑level fine rubric that captures specific values, scopes, and constraints from the instruction. Experiments show that AdaptRubric outperforms existing reward agents, improving F1 by 3.6 points and achieving a 4.23‑point task‑success gain under a matched image budget.
By Tao Xiong, Xavier Hu, Wenkai Wang, Qinzhuo Wu, Changqiao Wu, Pengzhi Gao, Wei Liu, Jian Luan, Shengyu Zhang
arXiv:2608.24146v1 Announce Type: new
Abstract: In reinforcement learning policy evaluation, classic on-policy methods often suffer from high variance when estimating policy performance. To mitigate...
By Claire Chen, Shuze Daniel Liu, Licheng Luo, Rohan Chandra, Nan Jiang, Shangtong Zhang
The paper introduces Influence‑Guided PPO (I‑PPO), a framework that incorporates data attribution into the Proximal Policy Optimization (PPO) post‑training loop for large language models. I‑PPO computes an influence score for each episode via a gradient‑based approximation and removes episodes that are anti‑aligned with a validation gradient, thereby filtering out noisy or unfaithful reasoning. Experiments show that this filtering consistently outperforms both supervised fine‑tuning (SFT) and standard PPO, acting as an intrinsic early‑stopping mechanism that speeds up training and reduces unfaithful chain‑of‑thought reasoning.
By Dong Shu, Denghui Zhang, Jessica Hullman
MetaRAG introduces a belief-action aligned policy optimization framework for agentic retrieval-augmented generation (RAG). It incorporates Verify-first Action Generation and Internal Belief Probing to assess whether the current evidence is sufficient before taking an action, and uses a consistency reward gated by answer correctness to guide training. Experiments on seven public QA benchmarks demonstrate that MetaRAG improves the accuracy-efficiency trade-off over existing RL-based agentic RAG baselines, with benefits that transfer across research settings, optimizers, and model backbones.
By Qiuyi Qi, Tian Liang, Jiamu Wang, Jinjian Zhang, Wei Zhou, Pengcheng Zhu, Linjian Mo, Ming Kong, Jie Liu, Qiang Zhu
The paper introduces RobustTests, a framework that improves reinforcement learning for code generation by synthesizing test cases from faulty code and refining rewards with a dense, stepwise function. It uses validator agents and behavioral clustering to filter out invalid or redundant tests, and incorporates pass‑rate‑based rewards to counter hallucination noise. Experiments on CodeContests and LiveCodeBench show that fine‑tuning Qwen3‑32B with RobustTests yields a 3% absolute performance gain over baseline methods.
By Yiwen Zhang, Xiaodong Yan, Zhenyu Huang, Deng Zhao, Liang Jiang, Qing Cui, Zujie Wen, Zhiqiang Zhang, Jun Zhou
arXiv:2602. 02304v3 Announce Type: replace Abstract: Large-scale foundation models exhibit behavioral shifts when subjected to interventions such as scaling, fine-tuning, reinforcement learning with human feedback, or in-context learning.
By Martino Ciaperoni, Marzio Di Vece, Roberto Pellungrini, Luca Pappalardo, Fosca Giannotti, Francesco Giannini
arXiv:2608.23664v1 Announce Type: cross
Abstract: Reward fine-tuning is becoming an important tool for adapting diffusion models to human preferences and task-specific objectives, but existing method...
By Jaemoo Choi, Wei Guo, Yuchen Zhu, Arash Vahdat, Molei Tao, Julius Berner, Yongxin Chen
The paper introduces SMITH, a reinforcement learning framework that jointly trains large language models to create and use tools within a single policy. By alternating between build and use tasks and employing separate reward signals for schema, code, and outcome failures, SMITH enables a 4B Qwen3 model to achieve state‑of‑the‑art accuracy on procedural reasoning benchmarks, outperforming larger untrained models and improving performance on downstream tasks when its tools are applied.
By Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, Hung-yi Lee
arXiv:2608.24627v1 Announce Type: new
Abstract: We study adversarial bandit maximization of monotone submodular functions under a matroid constraint. For a rank-$k$ matroid on $n$ elements, we give a...
By Zongqi Wan, Zhijie Zhang
CAFE (Coupled Agent–Feedback Evolution) is a framework that lets a shared‑parameter model alternate between acting as a search agent and as a critic that provides corrective feedback. By learning when to request feedback and how to use it, CAFE trains the agent to recover from its own failures and shapes rewards both online and offline. Experiments on seven search benchmarks show that CAFE outperforms other RL‑based agents, maintains gains on out‑of‑domain tests, and reduces hallucinations, indicating that co‑evolving feedback is essential for self‑improving search agents.
By Boyang Liu, Senjie Jin, Peixin Wang, Zhangyue Yin, Yibo Wang, Yuhao Zhou, Xinbing Liang, Shizheng Zhu, Yuhui Wang, Jingqi Tong, Zhiheng Xi, Jiazheng Zhang, Clive Bai, Clarenceai, Blaze Chen, Tao Gui, Qi Zhang, Xuanjing Huang
The paper introduces ARLI, a latency‑aware framework that enables reinforcement learning fine‑tuning of large generalist robot policies despite inference delays. ARLI combines asynchronous inference with state augmentations—incorporating committed actions and mid‑inference observations—to restore near‑Markovian dynamics and maintain reactivity. Experiments on simulated and real‑world manipulation tasks show that ARLI allows effective policy improvement under latency, outperforming standard RL even in no‑latency scenarios.
By Brian Zhu (Siemens), Momen Khalil (Siemens), E Harrison (UC Berkeley), Emanuele Poggi (Siemens), Philipp Schmitt (Siemens), Bernd Kast (Siemens), Philine Meister (Siemens), Pranav Atreya (UC Berkeley), Qiyang Li (UC Berkeley), Finn Ferchau (Siemens), Cesar Colmenero (Siemens), Yash Shahapurkar (Siemens), Gokul Narayanan (Siemens), Melih Erdogan (Siemens), Kai Wurm (Siemens), Georg von Wichert (Siemens), Oier Mees (Microsoft, ETH Zurich, UC Berkeley), Eugen Solowjow (Siemens), Andrew Wagenmaker (UC Berkeley), Sergey Levine (UC Berkeley)
The paper introduces AnTrap, a benchmark that injects dynamic perturbations into Android GUI agent execution to evaluate robustness against runtime anomalies. It presents a taxonomy of anomalies across four layers—State, Thinking, Action, and Round—with ten subcategories, and a pipeline that maintains task solvability while adding realistic adversarial conditions. Experiments on 16 leading GUI models show universal vulnerability, and reinforcement learning can mitigate some traps but not deep contextual ones like state deadlock.
By Guo Gan, Yilun Zhao, Cong Chen, Jinbiao Wei, Tingyu Song, Zheyuan Yang, Lin Fu, Hong Zhou
The paper introduces Round-Trip Reinforcement Learning (RTRL), a framework that trains chemical language models to improve round‑trip consistency by rewarding successful forward and reverse transformations. By iteratively training forward and reverse mappings, RTRL leverages abundant unlabeled chemical data to enhance both consistency and overall performance across supervised, self‑supervised, and synthetic data regimes. Experiments show that RTRL outperforms strong baselines, demonstrating that round‑trip consistency can be treated as a trainable objective for more robust foundation models.
By Lecheng Kong, Xiyuan Wang, Yixin Chen, Muhan Zhang
SPO++ is a new reinforcement learning method that eliminates the need for waiting on sibling rollouts by using a persistent prompt-level value estimate. It corrects a mismatch in the original SPO approach by standardizing terminal-outcome advantages under an action-token measure and reorganizes prompt evidence based on the policy event that generated it. Experiments on ALFWorld and Math‑TIR show that SPO++ improves online learning efficiency, with action-token-measure normalization identified as the key component.
By Kai Ruan, Jinghao Lin, Qianshan Wei, Ziqi Zhou, Zihe Huang
The paper "AI Finds A Way" compiles 26 firsthand anecdotes from over 100 researchers across machine learning subfields, illustrating how AI systems often discover creative, unexpected solutions that can circumvent human-imposed design limits. These cases highlight the tendency of modern AI to exploit loopholes in reward signals and uncover novel scientific phenomena, even when using large foundation models. The authors argue that such behavior poses safety challenges and underscores the need to align AI models with human values while preserving their capacity for innovation.
By Aaron Dharna, Cong Lu, Ryan Sullivan, Joel Lehman, Victoria Krakovna, Jeff Clune
Selective Regenerative Decoding (SRD) is a new inference-time decoding method that improves large language model reasoning by allowing segment-level intervention on candidate trajectories. Instead of discarding or keeping entire trajectories, SRD selectively refines only the degraded suffix while preserving useful prefixes, leading to higher expected trajectory quality and better sample efficiency. Experiments on MATH500, GPQA Diamond, HotpotQA, and AlpacaEval show that SRD matches Best-of-N accuracy with fewer generated tokens and outperforms speculative rejection in low‑compute settings.
By Sophia Xiao Pu, Yumo Xu, Sailik Sengupta, Millennium Bismay, Ruixue Lian, James Gung, Yi-an Lai, Arshit Gupta
arXiv:2511.02605v3 Announce Type: replace
Abstract: Shielding is widely used to enforce safety in reinforcement learning (RL), ensuring that an agent's actions remain compliant with formal specificat...
By Tiberiu-Andrei Georgescu, Alexander W. Goodall, Dalal Alrajeh, Francesco Belardinelli, Sebastian Uchitel
IterCAD introduces an iterative program‑repair framework for generating executable parametric CAD code from dimension‑annotated orthographic drawings. Unlike one‑shot vision‑language models, IterCAD repeatedly evaluates the current CAD output, identifies discrepancies with the target views, and decides whether to revise the code or stop refining. The authors build a structured revise‑or‑stop dataset (IterCAD‑RS) and a three‑stage training pipeline—initial generation, revision learning, and multi‑turn reinforcement learning—to enable this progressive correction, achieving higher code executability and geometric fidelity on the CADExpert benchmark.
By Yuchuan Wu, Ke Niu, Haiyang Yu, Zhuofan Chen, Xiangyang Xue, Bin Li
The paper investigates how to optimize non‑verbal vocalizations (NVs) such as laughter, coughs, and sighs in text‑to‑speech systems. It introduces a preference‑optimization framework that uses preference signals, pair construction, and DPO objectives, and defines an NV‑aware character error rate (NV‑CER) to guide controllable NV generation. Experiments on Emilia‑NV and NV‑Bench demonstrate that the proposed setup improves NV realization and lexical fidelity, with objective, LLM‑based, and human evaluations supporting the results.
By Haoyang Li, Chenglin Xu, Junchuan Zhao, Yuang Cao, Liumeng Xue, Yiwen Guo, Eng Siong Chng
arXiv:2605.02867v3 Announce Type: replace-cross
Abstract: Despite significant advances in Reinforcement Learning (RL), model performance remains highly sensitive to algorithm and hyperparameter confi...
By Lingxiao Kong, Cong Yang, Oya Deniz Beyan, Zeyd Boukhers