arXiv AI
From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 29
From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning
The paper introduces Q-Target Pretrained Transformers (QTPT), a method that replaces supervised behavior cloning with a Bellman-style Q‑target objective for in‑context reinforcement learning. QTPT retains the context‑conditioned Transformer architecture but learns to estimate action values using rewards and transitions from the context, rather than merely imitating offline actions. The authors provide theoretical analysis in stochastic linear bandits and finite‑horizon MDPs, demonstrating improved robustness to weak or suboptimal data, and empirically show gains over supervised pretraining on controlled RL benchmarks and extensions to D4RL Kitchen and AntMaze.
arXiv Machine Learning
Aug 4
Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning
arXiv:2608. 02034v1 Announce Type: new Abstract: Multi-step returns accelerate reward propagation in off-policy reinforcement learning, but couple the evaluation of each decision to the suboptimal logged actions that follow it, inducing a pessimistic bias that grows with the horizon.
arXiv AI
Jun 30
TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning
arXiv:2401. 11512v2 Announce Type: replace-cross Abstract: Identifying the most suitable variables to represent the state is a fundamental challenge in Reinforcement Learning (RL).
arXiv AI
Jun 2
Agentic Transformers Provably Learn to Search via Reinforcement Learning
arXiv:2606. 00183v1 Announce Type: cross Abstract: Tree search is a central abstraction behind many language-agent reasoning and decision-making tasks: agents must explore actions, remember failures, and backtrack toward promising alternatives.
arXiv Machine Learning
Oct 15
Provable Transformer Context Compression for In-Context Reinforcement Learning
arXiv:2610.20004v1 Announce Type: new Abstract: Transformers can perform in-context reinforcement learning without parameter updates. However, existing theoretical constructions often require context...
arXiv Machine Learning
Oct 7
Reinforcement Learning for Hierarchical Reasoning Rewards: Minimax-Optimal Rates with Transformers
The paper studies reinforcement learning (RL) for post‑training language models on reasoning tasks, focusing on a hierarchical reward structure where each component becomes relevant only after the previous ones are resolved. It demonstrates that a Transformer‑based actor–critic algorithm that alternates between KL‑regularized policy sampling, critic fitting, and policy updates achieves minimax‑optimal rates in query budget and regularization strength, and is optimal for a fixed number of prompts. In contrast, sampling from a fixed reference distribution, as used in offline reward modeling, only yields a logarithmic regret decay, highlighting the advantage of on‑policy exploration in concentrating on high‑reward regions.