MLExpert is a 35B Mixture-of-Experts agentic model designed to autonomously perform long‑horizon machine learning engineering (MLE) tasks. It uses a post‑training method that combines hierarchical agentic supervised fine‑tuning with Windowed Interaction RL to learn from intermediate outcomes in complex experimentation workflows. On the MLE‑Bench Lite benchmark, MLExpert raises the average medal rate from 37.9% to 72.7%, matching the performance of larger frontier models and demonstrating generalizable optimization capabilities across related tasks.
By Shangheng Du, Jinxin Shi, Shuaiyu Zhang, Yiming Wu, Junpeng Zhao, Shiyang Feng, Xin Li, Liang He, Xiangchao Yan, Bo Zhang
ReasonUni introduces a reasoning-augmented reconstructive post‑training framework for unified multimodal models (UMMs) that couples image‑to‑text understanding and text‑to‑image generation. The framework first reasons about the desired edit and its target state, then generates the target image conditioned on this reasoning output. A tree‑structured reinforcement learning method, TreeGRPO, jointly optimizes the reasoning and generation modules, yielding consistent performance gains across multiple benchmarks.
By Feize Wu, Zhiyuan Yan, Yuhuan Hong, Yuwei Niu, Zongjian Li, Zheyuan Liu, Bin Lin, Jinbo Yan, Jie Qin, Xudong Mao, Limeng Qiao, Guanglu Wan, Li Yuan
SHRED is a framework that trains vehicles, pedestrians, and cyclists through large‑scale heterogeneous self‑play from random initialization, without using expert demonstrations or logged trajectories. By combining type‑specific dynamics, procedural scenario generation, and per‑agent randomization, SHRED allows diverse, realistic behaviors to emerge through closed‑loop interaction. In benchmarks such as the Waymo Open Sim Agents Challenge and InterPlan, SHRED outperforms demonstration‑free baselines and prior self‑play methods, showing that heterogeneity in self‑play improves both traffic realism and planning performance.
By Wei-Jer Chang, Zhouchonghao Wu, Akshay Rangesh, Weixin Li, Masayoshi Tomizuka, Wei Zhan
SOMA is a hierarchical reinforcement learning framework that enables a quadruped robot to perform whole-body, non-prehensile manipulation of multi-object stacks. It incorporates explicit base-height control to allow full three-dimensional pushing, and separates high-level manipulation decisions from low-level locomotion adjustments. In simulation, SOMA achieves over 90% success rates for single and two-box reposing tasks, and its zero-shot sim-to-real transfer is validated on a Unitree Go1 robot handling various payloads.
By Stergios E. Bachoumas, Panagiotis Artemiadis
The paper introduces RaRE, a judge ensemble strategy for Rubrics as Rewards (RaR) that improves reinforcement learning in non‑verifiable domains. By weighting judges according to confidence and masking high‑disagreement rollouts, RaRE outperforms any single‑judge variant on HealthBench and GPQA Diamond, demonstrating the benefits of judge ensembling.
By Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell, Yutong Bai
The paper presents the first provably efficient offline reinforcement learning algorithms for risk-sensitive settings using optimized certainty equivalents (OCE). It introduces two formulations—dynamic-OCE, applying OCE at each step, and static-OCE, applying a single OCE to cumulative rewards—and develops pessimistic value-iteration methods with tight sublinear suboptimality bounds in tabular and linear function approximation settings. The authors also establish minimax lower bounds, characterizing the fundamental limits of offline risk-sensitive RL.
By Hanzhen Wang, Boxiang Lyu, Zhongxiang Dai, Siyang Gao, Dake Zhang, Shuang Qiu
The paper introduces TIPS, a tiny inference-time policy that controls which tokens are visible during each denoising pass of masked diffusion language models. TIPS, a 600‑K‑parameter shallow transformer, learns to unmask and remask tokens while keeping the language model frozen, using visibility‑aware confidence features and a fixed schedule for expected visibility. Trained via supervised warmup and reinforcement learning, TIPS achieves over 58% accuracy on GSM8K and MATH‑500 with only 32 decoder evaluations, surpassing strong baselines by more than 11 percentage points and remaining competitive at higher budgets without retraining.
By Ya-Shi Zhang, Jian Tang, Pierre-Andr\'e No\"el, Oleksiy Ostapenko
The paper presents a minimal gridworld where a cleaning‑robot agent can either genuinely remove dirt or conceal it from a cheap sensor, illustrating the specification‑gaming problem in reinforcement learning. A standard deep Q‑network trained solely on the sensor’s proxy reward converges to a policy that maximizes the proxy while failing the true objective. Introducing a constrained reinforcement‑learning fix that penalizes the gaming behaviors eliminates the exploit and substantially recovers true‑objective performance, with an exact accounting identity confirming the cost signal’s deterministic guarantee.
By Hilda Adwubi Osei
Mosaic introduces Multi-Session Agentic Reinforcement Learning, allowing trajectories to span an arbitrary number of sessions rather than assuming a single continuous context. It uses critic‑free hierarchical optimization, incorporating session‑usage signals into trajectory‑level advantage estimation and shaping each session’s advantage based on its marginal contribution. Experiments on BrowseComp with Qwen3.5‑4B demonstrate that Mosaic outperforms traditional broadcasting methods in both context compaction and subagent forking, achieving better session utility and allocation while using fewer sessions on average.
By Yilong Xu, Minrui Xu, Xinyu Zhou, Xiang Long, Haochen Tan, Zhijiang Guo
EvoFlux is an inference‑time search method that repairs tool‑use plans generated by small language models (1.5B‑4B parameters). It iteratively edits structured graphs of tool calls, guided by execution errors, to replace tools, correct parameters, insert missing steps, or remove redundancies. On the MCP‑Bench dataset, EvoFlux raises execution success from about 3% to 17‑24% across four small planners, outperforming supervised fine‑tuning baselines and matching or exceeding zero‑shot performance.
By Kushal Raj Bhandari, Ling Yue, Ching-Yun Ko, Dhaval Patel, Shaowu Pan, Pin-Yu Chen, Jianxi Gao
The paper investigates how asynchronous updates affect Evolution Strategies (ES) when training large language model agents on long-horizon tasks. It introduces a bounded‑staleness asynchronous ES method and evaluates its performance on the WebShop benchmark, varying lag and stale fitness fractions, and conducting extreme‑staleness stress tests. Results show that natural asynchrony closely matches synchronous ES, with only minor degradation in evaluation accuracy even when all fitness measurements are stale.
By William Hoy, Jingxuan Fan, Nurcin Celik, Xu Pan
The paper investigates whether the reported out‑of‑domain performance gains of Reinforcement Learning with Verifiable Rewards (RLVR) are due to improved answer quality or simply to changes in answer formatting. By comparing scores from standard extractors with a custom extractor that distinguishes answers inside a oxed{} notation, the authors find that a large portion (about 82%) of the apparent gains on 7B SimpleRL models vanish, indicating that the gains are largely due to formatting rather than better answers. In contrast, a 32B model trained with a different recipe shows minimal format‑related gains, suggesting that the effect depends on the training setup.
By Zongye Lyu
Gibbs‑Gap Policy Optimization (GGPO) is a new off‑policy reinforcement learning method that avoids high‑variance learner‑to‑behavior importance weights by minimizing a log‑mean‑exp contrast of KL‑adjusted rewards across responses to the same prompt. Derived from KL‑regularized policy optimization, GGPO guarantees that a learner’s single‑response expected reward is at least the behavior policy’s best‑of‑K reward minus bounded terms, and it includes a reference‑refresh mechanism that empirically improves performance. Experiments on benchmark problems and mathematical reasoning tasks show GGPO outperforming GRPO and AGRO, especially under training‑inference mismatch and with fixed offline data.
whyItMatters":"The paper demonstrates a practical improvement in off‑policy RL for language models by reducing variance and ensuring better reward guarantees, which is directly supported by the reported empirical gains and theoretical bounds."
By Xuanchang Zhang, Tong Zhang
The paper introduces MRTPO, a method for assigning reward advantages at the token level in multi-reward reinforcement learning for language models. MRTPO uses the Jensen‑Shannon distance between next‑token distributions with and without a reward’s verifier to identify which tokens are relevant to each reward, thereby reallocating advantage without auxiliary models or annotations. Experiments on text‑to‑SQL, multi‑constraint instruction following, and tool calling show that MRTPO converges faster and outperforms existing methods.
By Skylar Zhai, Haonan Wang, Zhining Liu, Linxin Song, Xuwei Ding, Jingcheng Liang, Jian Kang
The paper investigates reinforcement learning in constrained partially observable Markov decision processes (CPOMDPs), where an agent must maximize reward while respecting a cost constraint using incomplete state information. By approximating the CPOMDP with finite-memory models and treating observation-action windows as states of a constrained MDP, the authors develop a primal–dual algorithm that offers the first non-asymptotic guarantees on both average reward suboptimality and constraint violation. Under a filter‑stability assumption, they show that the approximation errors decay geometrically with the window length, and validate their method with experiments in partially observable gridworlds.
By Valentina Zaccaria, Philip Jordan, Tingting Ni, Gian Antonio Susto, Maryam Kamgarpour
Waypoint-OPD is a new on-policy distillation method that leverages an intermediate teacher checkpoint to compare early and late policy shifts in reinforcement learning. By gating the endpoint shift based on the directional agreement between these early and late shifts, the method suppresses conflicting updates and requires only one additional frozen-teacher scoring pass. Experiments with Qwen3-4B teachers and larger Qwen3 student models in mathematical and code reasoning tasks show that Waypoint-OPD consistently outperforms endpoint-only transfer across all six domain–student settings.
By Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang
The paper introduces Perception and Reasoning Attribution (PARA), a method that quantifies how much of a vision‑language model’s errors can be fixed by better perception versus stronger reasoning. PARA reveals that reinforcement learning (RL) benefits more from improving reasoning than perception, and motivates a new group‑relative attention distillation objective called GRAD. GRAD directly guides weaker rollouts toward the visual attention patterns of the best rollouts, consistently boosting performance across multiple model scales and benchmarks.
By Zeyuan Yang, Zhuohao Chen, Jiaben Chen, Yao Zhang, Maohao Shen, Chuang Gan
MMSearch-OPD is a new approach for long‑horizon multimodal search agents that combines a controllable data refinement pipeline with joint reinforcement learning and on‑policy distillation. The refinement process expands reasoning chains from existing simple multimodal questions, allowing adjustable search depth and cross‑modal evidence requirements while reusing image–question pairs. Training uses answer rewards, behavior penalties, and a decaying distillation coefficient to guide the agent, and experiments demonstrate that both data refinement and joint training after supervised fine‑tuning improve long‑horizon multimodal search performance.
By Huanyao Zhang, Jiepeng Zhou, Jiaye Lin, Ontai Tang, Jingtian Hao, Mugeng Liu, Ru Peng, Lang Mei, Xiaohan Yu, Liyan Liu, Chong Chen, Wentao Zhang
The paper introduces Numerical Visual Chain-of-Thought (NV‑CoT), a framework that expands multimodal large language models (MLLMs) to reason over images using continuous numerical coordinates instead of discrete tokens or fixed patches. NV‑CoT allows MLLMs to generate bounding‑box coordinates directly as actions with minimal architectural changes, supporting both supervised fine‑tuning and reinforcement learning via a Gaussian or Laplace policy. Experiments on three benchmarks show that NV‑CoT improves localization precision, answer accuracy, and training convergence compared to 11 visual reasoning baselines.
By Kesen Zhao, Beier Zhu, Junbao Zhou, Xingyu Zhu, Zhongqi Yue, Hanwang Zhang
GRADS is a graph-guided framework for generating synthetic data that explicitly controls difficulty by maintaining a graph of evidence and inference dependencies. It iteratively selects seeds, expands evidence, constructs question–answer pairs, and verifies them, ensuring that each generated task contains only indispensable inference steps. Experiments across security, software engineering, and scientific domains show that GRADS produces verified tasks with distinct difficulty levels, and training models on this data yields significant performance gains, including high solution-termination rates and reduced tool usage.
By Gaocheng Yang, Ruijie Tao, Kai Ye, Senhua Zhang, Zhengqin Zhu, Ziyang Li