The paper introduces SORL, a framework that stabilizes off‑policy reinforcement learning for long‑horizon large language model agents. It identifies two key instability sources—token‑level policy granularity mismatches and high‑variance off‑policy updates—and proposes turn‑level importance sampling and clipping‑triggered normalization to align optimization with multi‑turn interactions. Two instantiations, SO‑PPO and SO‑GRPO, are evaluated on open‑domain, multi‑hop, and medical QA benchmarks, as well as on asynchronous RL for mathematical reasoning, showing improved robustness without the need for early stopping or heuristic tuning.
By Chenliang Li, Adel Elmahdy, Alex Boyd, Zhongruo Wang, Siliang Zeng, Alfredo Garcia, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Mingyi Hong
arXiv:2603.23184v2 Announce Type: replace-cross
Abstract: Despite the success of reinforcement learning from human feedback (RLHF), existing reward modeling methods largely rely on explicit feedback,...
By Hao Wang, Haocheng Yang, Licheng Pan, Lei Shen, Xiaoxi Li, Yinuo Wang, Zhichao Chen, Yuan Lu, Haoxuan Li, Zhouchen Lin
arXiv:2605.20740v2 Announce Type: replace-cross
Abstract: Large language models (LLMs) have emerged as flexible regressors capable of predicting real-valued quantities from heterogeneous inputs. Yet...
By Jungsoo Park, Hyungjoo Chae, Ethan Mendes, Jay DeYoung, Varsha Kishore, Wei Xu, Alan Ritter
arXiv:2607.02092v4 Announce Type: replace-cross
Abstract: Reinforcement learning can improve vision-language-action (VLA) policies beyond supervised fine-tuning, although this typically involves furt...
By Liuhaichen Yang, Zhuang Jiang, Chenchao Sheng, Ningwei Bai, Qichen Yin, Hanbo Ma, Junkai Liu, Junkai Sun, Dongcheng Lyu, Yi Dong, Zezhi Tang
The paper introduces Best‑of‑N Guidance (BoNG), a method that embeds the Best‑of‑N sampling principle directly into the reverse diffusion process of generative models. By performing online BoN selection over denoising particles and using the current best particle as a guidance signal, BoNG steers the particle population toward higher‑reward regions during generation. Empirical results show that BoNG outperforms both standard BoN and SMC across 36 comparisons, achieving the best performance in 29 cases and supporting multi‑output generation with significant speedup.
By Richard Lee Kim, Yeongmin Kim, Gyuwon Sim, Taekyu Kim, Minsang Park, Il-chul Moon
The paper introduces self‑referenced social preferences, allowing agents to learn cooperative behavior without observing others’ private rewards. Each agent models its own reward, applies this model to observed transitions of other agents, and uses the resulting assessments to inform either the learning reward or policy updates. Experiments on three social‑dilemma environments show that agents can achieve cooperation and more equitable outcomes even when independent learners fail, with the best integration point depending on the type of social preference used.
By Mohamed Ayman Mohamed, Harshil Kotamreddy, Marcos Menon Jose
The paper introduces Self‑Retrospection Distillation (SRD), a method that uses post‑hoc trajectories to supervise foresight predictions made before an agent acts. SRD distills information from completed rollouts into trajectory‑blind predictions, enabling the agent to anticipate useful actions and avoid pitfalls without requiring explicit foresight at inference time. Experiments on ten tool‑integrated reasoning and long‑horizon tasks show that SRD improves performance by up to 24.2 percentage points, especially when reward signals are scarce or uniform across rollouts.
By Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li, Siming Zhang, Jiaxin Zhang, Jixuan Chen, Fang Wu, Pan Lu, Silvio Savarese, Julian McAuley, Chien-Sheng Wu
arXiv:2610.07226v1 Announce Type: cross
Abstract: ``What are the irreducible conditions that are sufficient to produce an outcome?'' is one of the most common questions that recur across computation...
By T. Y. Tsui, Zihao Ye, Pengxiang Cai, Yanchao Li, Yuqiang Li, Zhehong Ai
Humanoid loco-manipulation of large, heavy objects demands forceful interaction across the entire body. However, such payloads shift a humanoid's center of mass and impose sustained loads across the u...
Enterprise generative AI applications require robust safety mechanisms that can accommodate diverse risk postures, evolving policies, and varying latency constraints. Current guardrail solutions often...
Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning the...
Scientific research often begins by synthesizing ideas from a set of related papers to identify gaps and formulate new directions. However, training language models to perform this form of literature-...
Off-policy evaluation (OPE) for contextual bandit policies becomes challenging when action-level importance weighting incurs excessive variance. Doubly robust (DR) estimation remains unbiased under co...
Agentic reinforcement learning (RL) disaggregates training from rollout, so each policy update must reach the rollout clusters before the next batch. Transferring a full 1T checkpoint for such weight...
In this work, we consider decision-focused learning (DFL) for a Markov decision process (MDP), where existing methods differentiate through the KKT conditions of the Bellman equation and require solvi...
Diffusion language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) language models, offering flexible token-update orders and parallel decoding. Recent dLLMs are often in...
Deep Reinforcement Learning policies can produce nonsmooth action oscillations that hinder deployment on physical robots. Existing architectural and penalty-based approaches seek spatial smoothness by...
As part of learning to play complex games, human players develop develop abstractions for concepts and strategies of gameplay consistent with game rules to improve their performance. These concepts ar...
The paper introduces DUCB-OGD, an algorithm that couples a Discounted Upper‑Confidence‑Bound sampler with Online Gradient Descent to address dynamic minimax regret in robust large‑language‑model post‑training. It operates under instantaneous mini‑batch‑only bandit feedback, tracking worst‑source performance without re‑evaluating historical data. Experiments on fine‑tuning, preference optimization, and reinforcement learning demonstrate that DUCB‑OGD improves worst‑group robustness with negligible computational overhead.
The paper introduces Guidance‑TTT, a method that separates strategic planning from execution in test‑time training for large language models. A small guidance model is trained at test time to propose high‑level changes, while a frozen, larger execution model implements these changes, reducing the cost of maintaining gradients and optimizer states. Guidance‑TTT achieves strong results across four domains—combinatorial optimization, heuristic programming, machine learning, and GPU kernel optimization—outperforming prior work and matching state‑of‑the‑art leaderboard scores.