arXiv Machine Learning

Policy Gradient Steering: Interventions from Behavioral Objectives

arXiv:2607. 27574v1 Announce Type: new Abstract: Activation steering has emerged in large language models as a lightweight alternative for dynamically changing a model's behavior at inference time.

arXiv Machine Learning
Sep 11

UBCL: A Reinforcement Learning Framework for Controllable and Diverse Player Behaviors

The paper presents UBCL, a reinforcement learning framework that generates controllable and diverse player behaviors without using human gameplay data. By defining behavior in an N‑dimensional continuous space and training a single PPO‑based multi‑agent policy with target behavior vectors, the method learns how actions affect behavioral statistics such as aggressiveness, mobility, and cooperativeness. Experiments in a custom Unity multiplayer game demonstrate that UBCL achieves greater behavioral diversity than a win‑only baseline and accurately matches specified behavior vectors across a range of targets.

By Atahan Cilan, Atay \"Ozg\"ovde
arXiv AI
Sep 18

UnifiedPlayers: Enhance Tool-Integrated Reasoning in Agentic Reinforcement Learning

UnifiedPlayers is a cooperative framework that jointly adapts planning, execution, and evaluation for tool-integrated reinforcement learning agents. It consists of a Planning Player that generates tasks, an Execution Player that creates multi-turn trajectories with Python tool calls, and an Evaluation Player that builds executable verifiers, all coordinated by role‑specific rewards under GRPO. The approach outperforms prior baselines on mathematical and general reasoning benchmarks and yields a verifier with high adversarial detection accuracy and more discriminative reward signals.

By Wenjie Liao, Liangjie Zhao, Zehong Cao
arXiv Computation and Language
Sep 21

ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL

ArenaFlow is a hierarchical credit propagation framework designed to improve reinforcement learning for open-ended agent tasks. It uses tournament-based relative ranking to generate trajectory-level rewards and structured reflective evaluation to identify pivotal success steps, reusable strategy skills, and skill usage attribution. The framework propagates advantages to high-confidence steps and maintains a global skill memory, enabling more targeted optimization and reusable skill priors for future exploration.

By Qiang Zhang, Ruixue Ding, Fanrui Zhang, Xi Chen, Boli Chen, Shihang Wang, Yinfeng Huang, Yi Zheng, Pengjun Xie, Kaipeng Zhang, Jiawei Liu, Zheng-Jun Zha
arXiv AI
Sep 23

Synthesizing Reactive Character Behaviors for Continuous Games via Programmatic Policy Search

The paper introduces a method for creating reactive character behaviors in continuous games as compact, human‑readable programs. It searches over a domain‑specific language that uses reactive geometric decisions and higher‑order constructs to discretize continuous behavior space, while eliminating redundant program forms through synthesis antipatterns. The approach, called agentic sketching, combines bottom‑up symbolic enumeration with top‑down guidance from a coding agent, and outperforms either technique alone on a benchmark of 14 continuous games.

By Maxim Gumin, Hsueh-Ti Derek Liu, Victor Zordan, Daniel Ritchie
arXiv AI
3d ago

Conditional Generation of Creative Chess Puzzles with Diffusion Models

The paper presents a method for generating creative chess puzzles using masked diffusion models that can be conditioned on tactical themes and partial board positions. It introduces an auxiliary best‑move prediction task that boosts solution uniqueness by 11.6% and theme‑conditioning accuracy by 2.5%. A reinforcement learning framework further increases the yield of unique, theme‑matching puzzles by 89.1%, and the authors release open‑weights models for the community.

By Aatu Selkee, Severi Rissanen, Xidong Feng, Tom Zahavy, Eric Malmi
arXiv AI
Aug 19

PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs

PlanPO introduces a group planning-aware policy optimization method for multi-turn agentic large language models, addressing the issue of advantage collapse caused by treating all successful trajectories equally. By incorporating coarse-to-fine advantage signals that reflect differences in trajectory and turn lengths, PlanPO encourages agents to learn generalizable planning and generation behaviors. Experiments show a 27.2% average improvement over GRPO on benchmarks such as ALFWorld, WebShop, and SciWorld, with minimal extra training cost.

By Dayang Liang, Liyuan He, Xuan Feng, Shuxin Li, Bo An, Yunlong Liu