The paper investigates the problem of sharing a single critic across multiple parallel environments in reinforcement learning. It shows that when environments assign different expected returns to the same state, a shared critic must reconcile conflicting value targets, which can distort advantage estimates and misguide policy updates. The authors propose a simple fix—providing the critic with the environment index—demonstrating through bandit models and experiments on CartPole, MuJoCo, BipedalWalker, and 16 Procgen games that this conditional critic stabilizes learning and boosts returns, achieving a 40.8% improvement in aggregate normalized return on unseen levels.
By Zhenya Liu, Yang Meng, Zhuokai Zhao, Xuefeng Liu, Yuxin Chen
The paper explores a runtime strategy-selection framework where a large language model (LLM) guides a pre‑trained reinforcement learning (RL) policy for non‑player characters (NPCs) in a Unity combat game without altering the underlying policy. Five NPC agents sharing a PPO policy were compared in a baseline setup and an LLM‑augmented setup, where a locally hosted Mistral 7B model assigns one of four tactical tags every five seconds based on live game state. Across 600 episodes against three scripted opponents, the LLM‑augmented agents more than doubled their win rate against a Balanced opponent, improved performance against an Evasive opponent, but struggled against an Aggressive opponent due to over‑reliance on encirclement; analysis of 2,430 strategy selections revealed limited zero‑shot differentiation with the model favoring Surround in 83.8% of cases.
By Hrithika Deepu Nair, Kayvan Karim
The paper investigates whether a proximal policy optimisation (PPO) agent can learn to act as a broker in a continuous‑time broker‑trader game that has an analytically solved optimal strategy. By discretising the continuous‑time reward and validating the implementation, the authors show that a PPO agent with a feed‑forward neural network can approximate the reference action when there is no uninformed order flow, but struggles under stochastic uninformed flow, with critics failing to rank actions reliably. The study further demonstrates that a causal certainty‑equivalent controller based on observable history outperforms PPO under partial information, and that freezing the analytical policy and fine‑tuning PPO after a change in execution cost yields a measurable improvement.
whyItMatters:"The analytical solution serves both as a diagnostic benchmark for RL performance and as a practical starting policy that can be adapted to changing market conditions, illustrating how theoretical finance models can guide and improve reinforcement learning in complex financial control tasks."
By Siu Tung Wong (Institute of Finance and Technology, University College London), Carlo Campajola (Institute of Finance and Technology, University College London, UZH Blockchain Center)
The paper presents a constrained command‑conditioned Proximal Policy Optimization (PPO) executor for the MicroRTS real‑time strategy environment, enabling a single execution policy to follow diverse strategic commands. A Thompson‑sampling bandit strategist selects command tuples based on in‑game observations of the opponent’s strategy, rather than relying on opponent identity. In controlled experiments, the strategist‑executor system outperforms a flat PPO baseline against three of the four strongest opponents on a training map, including two held‑out opponents, while showing no significant difference against the remaining opponents.
By Nick Leenders, Roy Lindelauf, Joost van Oijen, Boris Cule
The paper proposes CANOPY, a minimalist reinforcement learning protocol that addresses two common pitfalls—signal starvation and policy drift—in outcome‑only RL for long‑horizon interactive tasks. By scaling same‑task exploration, keeping updates on‑policy, and anchoring updates with KL divergence, CANOPY enables a Qwen3‑14B agent to achieve top leaderboard results on the AppWorld coding benchmark without auxiliary supervision or elaborate scaffolding. The approach also improves performance on SWE‑bench for a Qwen3.5‑9B model.
By Liming Pu, Xiaoxia Li, Yifu Liu, Teng Cao, Bin Yang
StraTA introduces Strategic Trajectory Abstraction, a framework that samples a compact strategy from the initial task state and conditions subsequent actions on that strategy, training strategy generation and action execution jointly with a hierarchical GRPO-style rollout design. The method enhances exploration and credit assignment over long horizons by incorporating diverse strategy rollouts and critical self-judgment. Experiments on ALFWorld, WebShop, and SciWorld demonstrate that StraTA consistently improves sample efficiency and final performance, achieving success rates of 93.1% on ALFWorld, 84.2% on WebShop, and a 63.5% overall score on SciWorld, surpassing frontier closed‑source models.
By Xiangyuan Xue, Yifan Zhou, Zidong Wang, Shengji Tang, Philip Torr, Wanli Ouyang, Lei Bai, Zhenfei Yin
arXiv:2609.38881v1 Announce Type: new
Abstract: Real-time strategy (RTS) games require agents to coordinate economic development, production and construction, base defense, unit organization, and att...
By Xinhe Tian, Xiaoyue Zhang, Ziyou Zhang, Jiacheng Li, Xiaoqiang Jin, Qianchuan Zhao, Gaochen Cui
arXiv:2610.08402v1 Announce Type: new
Abstract: Multi-turn LLM agents often receive sparse task feedback across several interactions, while generating each response token by token. This creates two r...
By Jiaju Chen, Min Yang, Jinghua Piao, Xiaochong Lan, Xu Xia, Xiangnan He, Yong Li
arXiv:2608. 06735v1 Announce Type: new Abstract: Reinforcement learning (RL) has achieved strong results in improving large language models (LLMs) on tasks with stationary, verifiable rewards, such as mathematical reasoning and code execution.
By Senhao Wang, Chenghao Cai, Haitao Hu, Mingxing Huang, Xingguang Wang, Wenhao Li, Zecheng Lin
arXiv:2608. 01425v1 Announce Type: cross Abstract: Training LLM-based multi-agent systems with multi-agent reinforcement learning is rapidly gaining traction, and a parallel line of work argues that such systems should be judged by their behavior, not only their reward.
By Yi Mao, Andrew Perrault
The paper introduces the World-Model Policy Arbiter (WMPA), a test‑time framework that selects among a portfolio of frozen goal‑conditioned policies by simulating each policy’s future trajectories in a learned world model and evaluating them with a shared value function. WMPA avoids frequent policy switching by committing to the chosen policy for a short interval and does not require retraining or task‑specific knowledge. On the OGBench benchmark, WMPA raises the macro‑average success rate from 44% to 58%, achieving significant improvements on 12 of 18 datasets, including large gains on cube‑double‑play and scene‑play.
By Junwei Quan, Evgenii Opryshko, Nicholas Rhinehart, Igor Gilitschenski
The paper investigates collusive behavior in a two‑player Almgren‑Chriss liquidation game using independent proximal policy optimisation agents. By training agents that learn a punitive mechanism, the study shows that when one agent deviates from the mean liquidation schedule, the other responds by accelerating liquidation, thereby negating the deviator’s gains while keeping its own payoff stable. The authors formalize checks confirming that punishment outweighs deviation gains and that behavioral changes are sufficient to offset losses, providing both behavioral and economic evidence of collusion.
By Christos Spyridon Koulouris, Carlo Campajola