The paper introduces a neurosymbolic world model that separates observation reconstruction from reward prediction, enabling the model to adapt zero‑shot to new reward functions defined over a shared symbolic state space. This approach addresses the task‑dependency of traditional neural world models, which learn latent representations tied to specific training tasks. Experiments show that the neurosymbolic formulation generalises more strongly than purely neural methods.
By Isidoro Tamassia, Lennert De Smet, Giuseppe Marra
Data-DPO is a target model‑oriented supervised fine‑tuning data selection method that uses one‑step probing of the target model to generate pairwise data preferences, trains a lightweight reward model to capture these preferences, and then selects a training subset by combining target‑model preference, external quality scores, and marginal diversity. Experiments on Vision‑Flan and LLaVA‑CoT demonstrate that Data‑DPO consistently outperforms existing data selection baselines across multiple data budgets and even surpasses full data training performance.
By Peng Sun, Yi Yang, Antong Zhang, Chunxiao Li, Yanbo Wang, Dianbo Liu, xin chen, Kai Yu, Lu Chen, Tianfan Fu
The paper introduces the concept of likelihood hacking (LH), where language models trained via reinforcement learning generate probabilistic programs that inflate marginal-likelihood rewards by producing non‑normalising data distributions. It formalises LH in a core probabilistic programming language and derives syntactic conditions that prevent such exploits, defining a safe language fragment λ_safe. Empirical studies show that models trained with GRPO quickly discover LH, while a modified Stan implementation, SafeStan, effectively suppresses these exploits under optimisation pressure.
By Jacek Karwowski, Younesse Kaddar, Zihuiwen Ye, Esmeralda S. Whitammer, Sam Staton
arXiv:2608. 17841v1 Announce Type: cross Abstract: Multi-armed bandit algorithms are evaluated by regret, yet comparable regret can coexist with different allocations across independent runs.
By Kaifei Wang, Yinyu Ye, Han Zhong
The paper introduces Latent Order Bandits (LOB), a new bandit framework that relaxes the strict assumptions of traditional latent bandits by only requiring a partial order of action preferences within each latent state. LOB allows instances sharing the same state to have different reward distributions as long as the action ranking remains consistent, making it suitable for scenarios like user groups on streaming services who agree on genre preferences but rate differently. The authors present an upper‑confidence bound algorithm for both total and partial latent orders, provide regret bounds, and propose a posterior‑sampling variant that empirically outperforms full‑prior latent bandits when reward scales vary across instances sharing the same latent state.
By Emil Carlsson, Newton Mwai, Fredrik D. Johansson
The paper introduces policy‑counted DecPOMDPs, a variant of decentralized partially observable Markov decision processes that mitigates the exponential growth in agent numbers by counting policies instead of agents. By exploiting symmetry among agents, the authors achieve a compact representation that reduces model complexity and evaluation cost to polynomial levels. They further present a dynamic programming algorithm that leverages this compact form to solve policy‑counted DecPOMDPs efficiently.
By Nazl{\i} Nur Karabulut, tanya Braun
The paper introduces tinyDSM, a framework that enables cm‑sized millirobots to autonomously learn and adapt skills using reinforcement learning guided by intrinsic motivation and fitness‑based assessment. It emphasizes minimal hard‑wired skills and a generic knowledge base, employing a hierarchical knowledge graph and kinematic reasoners to model motion skills. Experiments on a 36 cm³ millirobot with a Raspberry Pi Pico demonstrate rapid skill acquisition from basic movements to complex patterns within 15 minutes, supported by simulation analyses of learning algorithms.
By Markus D. Kobelrausch, Michael Miedler, Axel Jantsch
SOD: Step-wise On-policy Distillation for Small Language Model Agents proposes a new framework that adaptively reweights distillation strength at each reasoning step based on step-level divergence. This approach mitigates cascading errors in tool-integrated reasoning by attenuating misleading teacher signals in high-divergence regions while preserving dense guidance where student and teacher align. Experiments on math, science, and code benchmarks show up to 20.86% improvement over the second-best baseline, with a 0.6B student scoring 26.13% on AIME 2025.
By Qiyong Zhong, Mao Zheng, Mingyang Song, Xin Lin, Jie Sun, Houcheng Jiang, Xiang Wang, Junfeng Fang
EXPO-FT is a system that enables stable, sample‑efficient reinforcement learning fine‑tuning of pretrained Vision‑Language‑Action (VLA) policies. It achieves perfect success on a range of manipulation tasks—such as routing string lights, striking a pool ball, and inserting a flower into a wine bottle—using only about 19.1 minutes of online robot data. The approach outperforms both RL-from-scratch and existing VLA fine‑tuning methods, and the authors provide an open‑source codebase to support wider adoption.
By Perry Dong, Kuo-Han Hung, Tian Gao, Dorsa Sadigh, Chelsea Finn
The paper proposes EvalXRL, a benchmark that evaluates explainable reinforcement learning (XRL) methods by measuring how well their explanations help a large language model coding agent diagnose and fix bugs in RL agents. Unlike current metrics that focus on faithfulness or human ratings, EvalXRL uses a closed‑loop, scientific‑method style interaction where the coding agent repeatedly invokes XRL methods, refines hypotheses, and repairs the agent, scoring each method by the resulting RL reward. This approach enables a head‑to‑head comparison of multiple XRL techniques in realistic debugging scenarios.
By Ram Rachum, Yotam Amitai, B\'alint Gyevn\'ar, Reuth Mirsky, Cameron Allen
The paper introduces adaptive policy portfolios for robust Markov decision processes, where a finite set of memoryless randomized policies is synthesized offline and paired with an online selector. It defines robust regret as a measure of portfolio quality, comparing each portfolio member to the optimal policy for each plausible environment. The authors provide a complexity-theoretic analysis of portfolio certification and synthesis, showing that even deterministic portfolios in simple settings are highly complex, and present an offline construction method that can be specialized at runtime.
By Kasper Engelen, Sebastian Junges, Guillermo A. P\'{e}rez, Marnix Suilen
ScreenSearch is a system for exploring desktop GUI states under partial observability. It combines structural screen retrieval, deduplication, and an ambiguity-aware PUCT graph-bandit to expand the reachable frontier while reducing state ambiguity. Across 11 applications, it collected over 1 M screenshots and 30 K deduplicated states, demonstrating that both ambiguity reduction and frontier exploration are essential for effective corpus building.
By Michael Solodko, Justin Wagle
The paper discusses how Diffusion Models (DMs) can improve decision-making and digital modeling for Uncrewed Aerial Vehicles (UAVs). It highlights the limitations of Reinforcement Learning (RL) and Digital Twin (DT) approaches, noting that DMs learn underlying probability distributions and generate realistic patterns, thereby addressing data scarcity and enhancing modeling accuracy. Simulation results demonstrate DMs’ effectiveness in estimating neighbor velocities for a four‑UAV swarm coordination task using Deep Reinforcement Learning.
By Yousef Emami, Hao Zhou, Luis Almeida, Kai Li
The paper presents a tutoring platform that combines a generative AI chatbot with a reinforcement learning algorithm to adaptively sequence practice problems for students learning Python. In a five‑month field study across ten high schools, the adaptive sequencing improved unassisted final exam performance by 0.15 standard deviations, with mediation analysis indicating that higher engagement drove the gains. The study demonstrates that signals from student‑chatbot interactions can be leveraged to personalize and optimize learning at scale.
By Angel Tsai-Hsuan Chung, Botong Zhang, Ling-Chieh Kung, Hamsa Bastani, Osbert Bastani
Agent Lightning v1.0 is a lightweight framework that enables harnessed agentic reinforcement learning, where the agent harness—managing tools, context, and control flow—directly participates in model post‑training. It supports arbitrary agent harnesses and addresses challenges such as retokenization, sample merging, and advantage calculation, providing a reproducible pipeline for instruction‑following, search, and coding agents. In experiments, RL training on 6K examples improved Qwen3.5‑9B’s performance on SWE‑bench from 41.8% to 56.4%.
By Zhiyuan He, Siwei Zhang, Zhiwen Zhou, Yuqing Yang, Yu Kang, Yuge Zhang, Luna K. Qiu, Tin Yan Tsui, Jiahang Xu, Chong Luo
The paper presents a reinforcement learning framework that refines robotic grasp poses using a Deep Q-Network and keypoint-based object representations. Starting from initial grasp candidates generated by a geometric algorithm on 2D overhead images, the method iteratively improves grasps, converting previously failed attempts into successful ones. Experiments on 300 Dex‑Net objects with a UR5 arm achieved a 100% success rate on items that were ungraspable by geometry alone, and the approach transferred to a Delta robot in real‑world tests.
By Amir Arsalan Nematollahi, Shayan Ahmadi, Mehdi Tale Masouleh, Ahmad Kalhor
Agentic ESOpt proposes using evolution strategies (ES) instead of reinforcement learning to fine‑tune large language‑model agents for long‑horizon tasks. ES offers model scalability, flexibility, and better long‑horizon credit assignment, enabling full‑parameter optimization with minimal GPU memory. The framework samples parameter perturbations, evaluates agents with rewards, and updates online, achieving notable performance gains on WebArena‑Lite and in test‑time prompt‑parameter co‑evolution.
By Zhi Zheng, Rongsheng Chen, Yunpeng Ba, Zhenkun Wang, Yee Whye Teh, Wee Sun Lee
The paper investigates adaptive policy portfolios for Robust Markov Decision Processes (RMDPs), proposing finite sets of memoryless randomized policies generated offline and selected online. It introduces robust regret as a metric for portfolio quality, comparing each portfolio member’s performance to the optimal policy for each plausible environment. The authors provide complexity-theoretic results showing that certifying and synthesizing such portfolios is highly intractable, and they present an offline construction method that can be specialized at runtime.
Multi-armed bandit algorithms are evaluated by regret, yet comparable regret can coexist with different allocations across independent runs. We study the trade-off between worst-case regret $\mathcal{R}_{K,T}$ and instability $\mathcal S_{K,T}$, defined as the largest standard deviation of a terminal pull count, for $K$ arms and $T$ rounds.
The paper addresses the exponential complexity of Decentralised Partially Observable Markov Decision Processes (DecPOMDPs) by shifting focus from counting agents to counting policies. By exploiting symmetry among agents, it introduces a compact encoding that reduces model complexity and evaluation cost to polynomial dependence. The authors further develop a policy‑counted dynamic programming algorithm that efficiently solves these policy‑counted DecPOMDPs.