The paper introduces Q-Target Pretrained Transformers (QTPT), a method that replaces supervised behavior cloning with a Bellman-style Q‑target objective for in‑context reinforcement learning. QTPT retains the context‑conditioned Transformer architecture but learns to estimate action values using rewards and transitions from the context, rather than merely imitating offline actions. The authors provide theoretical analysis in stochastic linear bandits and finite‑horizon MDPs, demonstrating improved robustness to weak or suboptimal data, and empirically show gains over supervised pretraining on controlled RL benchmarks and extensions to D4RL Kitchen and AntMaze.
By Yichen Lin, Xuyuan Xiong, Xue Wang, Xiangfu Meng, Mike Mingcheng Wei, Tao Yao
arXiv:2608. 02034v1 Announce Type: new Abstract: Multi-step returns accelerate reward propagation in off-policy reinforcement learning, but couple the evaluation of each decision to the suboptimal logged actions that follow it, inducing a pessimistic bias that grows with the horizon.
By Abdelghani Ghanem, Mounir Ghogho
arXiv:2401. 11512v2 Announce Type: replace-cross Abstract: Identifying the most suitable variables to represent the state is a fundamental challenge in Reinforcement Learning (RL).
By Charles Westphal, Stephen Hailes, Mirco Musolesi
arXiv:2606. 00183v1 Announce Type: cross Abstract: Tree search is a central abstraction behind many language-agent reasoning and decision-making tasks: agents must explore actions, remember failures, and backtrack toward promising alternatives.
By Tong Yang, Yu Huang, Yingbin Liang, Yuejie Chi
arXiv:2610.20004v1 Announce Type: new
Abstract: Transformers can perform in-context reinforcement learning without parameter updates. However, existing theoretical constructions often require context...
By Yajie Zhao, Yingjun Mo, Zhiyong Wang, Bo Tang, Chao Xu, Fang Kong
The paper studies reinforcement learning (RL) for post‑training language models on reasoning tasks, focusing on a hierarchical reward structure where each component becomes relevant only after the previous ones are resolved. It demonstrates that a Transformer‑based actor–critic algorithm that alternates between KL‑regularized policy sampling, critic fitting, and policy updates achieves minimax‑optimal rates in query budget and regularization strength, and is optimal for a fixed number of prompts. In contrast, sampling from a fixed reference distribution, as used in offline reward modeling, only yields a logarithmic regret decay, highlighting the advantage of on‑policy exploration in concentrating on high‑reward regions.
By Naoki Nishikawa, Taiji Suzuki
The paper introduces BUMEX, a reinforcement learning exploration strategy that leverages a set of prior models containing the true transition kernel and reward function. By optimizing over this model set, the method derives upper and lower bounds on the Q‑function to guide exploration, providing theoretical guarantees of convergence to the optimal policy. When the model set follows a bounded‑parameter MDP structure, the optimization becomes convex, enabling finite‑time convergence under mild assumptions and demonstrating accelerated learning in simulations.
By J. S. van Hulst, W. P. M. H. Heemels, D. J. Antunes
The paper introduces Agentic In-Context Learning (Agentic ICL), a framework where fixed‑parameter Transformers learn from interactions that shape the context they receive. It models an interactive learner with state, update, and decision rules, covering active learning, bandits, and online reinforcement learning, and demonstrates that causal Transformers can approximate these algorithms over a finite horizon. Experiments show that Transformers can learn to execute recursions and that different reward signals produce distinct decision policies, with alignment between process supervision and outcome‑optimal actions influencing learned behavior.
By Shiguang Wu, Yaqing Wang, Quanming Yao
arXiv:2610.14988v1 Announce Type: cross
Abstract: The average-reward setting for reinforcement learning is challenging because its Bellman operator is known to contract in span only when the underlyi...
By Julien Roger, Sam Bodmer, Jian Qian, Benjamin Riviere
The paper introduces state abstractions that preserve the difference of Q‑functions for offline reinforcement learning, aiming to exclude irrelevant dynamics from rich state data. It proposes a dynamic generalization of the R‑learner that uses orthogonal estimation and sparse learning to estimate the Q‑function contrast, achieving faster convergence and consistency under a margin condition. Experiments on simulated and simulator‑augmented real data show variance reductions and demonstrate that the necessary information for sequential decision‑making can be smaller than that required for full state prediction.
By Defu Cao, Angela Zhou
The paper introduces Q-learning Penalized Transformer (QPT), a training–inference consistent framework for safe offline reinforcement learning. QPT trains a Transformer policy that generates actions conditioned on trajectory context and target return/cost while incorporating a Q-shaped penalty to balance safety, reward maximization, and behavior regularization. The method consistently outperforms strong baselines on 38 DSRL benchmark tasks and adapts robustly to varying constraint thresholds.
arXiv:2609.34426v2 Announce Type: replace
Abstract: This paper addresses the problem of safe offline reinforcement learning, which involves training a policy to satisfy safety constraints using an of...
By Shengchao Hu, Peng Wang, Jifeng Hu, Qiyang Zhou, Anning Hu, Li Shen, Ya Zhang, Dacheng Tao