The paper introduces the λ-hold controller, a minimal-task-reward approach inspired by the equilibrium-point hypothesis, to train a muscle-actuated skeletal model for human-like sprinting. By fixing each muscle’s EP threshold length λ over gait-phase intervals, the controller dramatically reduces the action space and the frequency of policy queries, enabling efficient exploration and learning within an hour of training. This method demonstrates that physiologically grounded control can produce realistic human motion in predictive musculoskeletal simulations.
By Jun Hyuk Lee, Chihyeong Lee, Jooeun Ahn
The paper proposes Novelty and Surprise Prioritized Experience Replay (NSPER) for image-based reinforcement learning, combining novelty to highlight underrepresented states and surprise to reveal gaps in the agent’s knowledge. An extended version, NSPER+R, also uses these signals as intrinsic rewards to enhance both replay quality and exploration. Experiments on DeepMind Control Suite tasks demonstrate that NSPER and NSPER+R accelerate training and improve convergence compared to existing methods.
By Hoda Yamani, Henry Williams, Bruce A. MacDonald
Efficient RLVR Scheduling via Graph-Structured Online Difficulty Estimation proposes a plug‑and‑play graph‑based online difficulty estimator for reinforcement learning with verifiable rewards (RLVR). The method constructs a difficulty‑aware sample graph using semantic and reasoning similarities, introduces latent difficulty states with a Potts prior, aggregates rollout outcomes with a state‑level Beta‑Binomial model, and updates these estimates online via a mean‑field variational algorithm. This framework can be integrated into sample‑selection and rollout‑allocation schedulers, enabling difficulty‑adaptive exploration without dedicated probing and achieving better performance across multiple base models, RL schedulers, and benchmarks.
By Zhizhao Liu, Zhiliang Tian, Xi Wang, Zhihua Wen, Yihang Xiong, Zhiquan Lai, Dongsheng Li
The paper introduces a framework for combining large language models (LLMs) with reinforcement learning (RL) by treating the LLM as a planner and the RL agent as a controller. It formalizes this hybrid setup as a Goal-Augmented Markov Decision Process and proves that using the LLM’s per‑state progress score as a bounded potential function preserves the optimal policy set, even if the LLM scores are inaccurate. The authors validate their theoretical result with numerical experiments on a small MDP, testing four potential configurations, including an adversarial case with a potential scaled twenty times the base reward.
By Christophe D. Hounwanou, John Emeka Eze, Ya\'e U. Gaba
The paper shows that fine‑tuning a large language model (LLM) with a debate framework—where a generator and a critic compete and a weaker LLM judge adjudicates—reduces reward hacking compared to standard reinforcement learning from AI feedback (RLAIF). In experiments on mathematics tasks, the debate approach keeps the judge’s performance stable, achieving a 45% higher peak validation accuracy than the RLAIF baseline and mitigating the rapid exploitation of judge errors. Additional findings indicate that weakening the judge speeds hacking unless countered by extra debate rounds, that debate can override misalignment prompts, and that word‑limit constraints on critiques help balance the game and prevent judge hacking.
whyItMatters":"The study demonstrates a practical method to curb reward hacking in RL‑based AI systems, addressing a key obstacle for safely scaling AI oversight."
By Zachary Kenton, Lili Janzer, Rory Greig, Tian Huey Teh, Kirill Tyshchuk, Jonah Brown-Cohen, Harri Edwards, Senthooran Rajamanoharan, Noah Y. Siegel, Natasha Jaques, Rohin Shah
The paper introduces a mesh‑free policy iteration framework that blends classical dynamic programming with physics‑informed neural networks (PINNs) to solve high‑dimensional, nonconvex Hamilton–Jacobi–Isaacs (HJI) equations. The method alternates between solving linear second‑order PDEs under fixed feedback policies and updating controls via pointwise minimax optimization using automatic differentiation. The authors prove local uniform convergence of the value function iterates to the unique viscosity solution under standard Lipschitz and uniform ellipticity assumptions, and demonstrate the approach’s accuracy and scalability in two‑, five‑, and ten‑dimensional stochastic games, outperforming direct PINN solvers.
By Hee Jun Yang, Minjung Gim, Yeoneung Kim
The paper proposes Instant Episode Repetition (IER), a method that immediately repeats action sequences from high‑reward episodes during training to boost sample efficiency. Unlike passive replay techniques, IER actively shapes data collection by re‑executing successful behaviors for a set number of subsequent episodes. Experiments on MuJoCo, the DeepMind Control Suite, and a robotic manipulation task show that IER improves learning performance over standard SAC, TD3, and self‑imitation baselines.
By Hoda Yamani, Yuning Xing, Koen van Rijnsoever, Bruce A. MacDonald, Henry Williams
The paper introduces a two-player zero-sum repeated game between a learner and nature that simultaneously captures Bayesian updating and an exact decomposition of exponential-weights regret. The game’s terminal payoff reflects the maximum gain a comparator can achieve given a fixed relative entropy from the prior, while the one-step constraint limits nature’s move by an information budget. The resulting regret splits into three precise components—per-round information loss, an additive retempering drift, and the comparator’s information relative to the prior—providing a unified framework that explains concentration phenomena, large-deviation bounds, and various learning methods such as bandits, posterior sampling, aggregation, and boosting.
By Akshay Balsubramani
The paper presents a tutoring platform that combines a generative AI chatbot with a reinforcement learning algorithm to adaptively sequence practice problems for students learning Python. In a five‑month field study across ten high schools, the adaptive sequencing improved unassisted final exam performance by 0.15 standard deviations, with mediation analysis indicating that higher engagement drove the gains. The study demonstrates that signals from student‑chatbot interactions can be leveraged to personalize and optimize learning at scale.
By Angel Tsai-Hsuan Chung, Botong Zhang, Ling-Chieh Kung, Hamsa Bastani, Osbert Bastani
The paper discusses how Diffusion Models (DMs) can improve decision-making and digital modeling for Uncrewed Aerial Vehicles (UAVs). It highlights the limitations of Reinforcement Learning (RL) and Digital Twin (DT) approaches, noting that DMs learn underlying probability distributions and generate realistic patterns, thereby addressing data scarcity and enhancing modeling accuracy. Simulation results demonstrate DMs’ effectiveness in estimating neighbor velocities for a four‑UAV swarm coordination task using Deep Reinforcement Learning.
By Yousef Emami, Hao Zhou, Luis Almeida, Kai Li
The paper introduces QWM, a framework that integrates world models with standard Q‑learning to perform test‑time search over imagined trajectories. By training the policy and value function solely on real transitions, QWM avoids compounding model bias while still benefiting from predictive search. Experiments on the Robomimic and LIBERO manipulation benchmarks show that QWM outperforms strong prior state‑of‑the‑art methods in both sample efficiency and performance.
By Perry Dong, Yueru Jia, Chelsea Finn, Dorsa Sadigh
LEGO-RL is a framework that connects native coding-agent harnesses with scalable policy‑gradient training without altering the harnesses’ internal flow. It achieves faithful optimization through in‑process LLM proxying, reliable execution via sandbox orchestration, and observable training with automated validation and a Live UI. Experiments show LEGO‑RL improves the Qwen3.5‑35B‑A3B model’s performance on three native harnesses while preserving high rollout‑training probability correlation.
By Yiming Du, Yuxin Jiang, Tao Yuan, Jianbo Dai, Shaowei Wang, Jierun Chen, Chaofan Tao, Xianzhi Yu, Lifeng Shang, Kam-Fai Wong, Xiaohui Li, Haoli Bai
PlanPO introduces a group planning-aware policy optimization method for multi-turn agentic large language models, addressing the issue of advantage collapse caused by treating all successful trajectories equally. By incorporating coarse-to-fine advantage signals that reflect differences in trajectory and turn lengths, PlanPO encourages agents to learn generalizable planning and generation behaviors. Experiments show a 27.2% average improvement over GRPO on benchmarks such as ALFWorld, WebShop, and SciWorld, with minimal extra training cost.
By Dayang Liang, Liyuan He, Xuan Feng, Shuxin Li, Bo An, Yunlong Liu
The paper introduces FACA, a Feedback‑Aware Credit Assignment method that aligns each agent reaction with the preceding user‑to‑user segment, computes a locally normalized reaction advantage, and adds it to the terminal outcome advantage without requiring an extra critic or rollout. Experiments show that FACA improves performance across nine domains, especially in Telecom, and maintains the same ordering of gains in zero‑shot benchmarks such as Pare‑Bench and Co‑Gym.
By Yiwen Zhao, Zhihao Wen, Yuchen Mao, Mingxuan Jiang, Yihao Hu, Pan Wang, Xin Zhang, Wei Wu
The paper evaluates how three large mixture‑of‑experts models (Alibaba, OpenAI, NVIDIA) can be fine‑tuned to reason in a low‑resource language, specifically Greek. Accuracy metrics show little change, but the authors uncover significant qualitative improvements: after supervised fine‑tuning, models reason in Greek on ~98% of items, with better grammaticality and retained general ability. Reinforcement learning with pre‑registered rewards further eliminates reasoning‑channel leaks and format skips, while the Greek‑reasoning habit remains robust to an accuracy‑only gradient.
By Ayoub Kirouane, Christos Petrocheilos
SignalReasoner evaluates reinforcement fine‑tuning methods for the Qwen2.5‑3B‑Base language model on graduate‑level signal processing math problems from the WirelessMATHBench‑XL benchmark. Two training paradigms are compared: direct reinforcement learning with verifiable rewards, and supervised chain‑of‑thought fine‑tuning followed by domain‑specific reinforcement learning. The study benchmarks three policy optimization algorithms—GRPO, GSPO, and GMPO—to determine if domain‑aware chain‑of‑thought initialization improves performance, achieving a best accuracy of 39.12% versus 12.37% for the untrained base model.
By Guozheng Sun
The paper introduces tinyDSM, a framework that enables cm‑sized millirobots to autonomously learn and adapt skills using reinforcement learning guided by intrinsic motivation and fitness‑based assessment. It emphasizes minimal hard‑wired skills and a generic knowledge base, employing a hierarchical knowledge graph and kinematic reasoners to model motion skills. Experiments on a 36 cm³ millirobot with a Raspberry Pi Pico demonstrate rapid skill acquisition from basic movements to complex patterns within 15 minutes, supported by simulation analyses of learning algorithms.
By Markus D. Kobelrausch, Michael Miedler, Axel Jantsch
The paper introduces WONDER, a radio world‑model‑based negotiation framework designed to optimize multi‑UAV coverage for rapid post‑disaster wireless restoration. WONDER employs a Joint‑Embedding Predictive Architecture to forecast the incremental radio impact of candidate UAV trajectories and uses multi‑round negotiation to sequentially commit trajectories while updating the context. Experiments in the RadioDynamics simulation environment demonstrate that WONDER outperforms six other methods, achieving a balanced score of 0.870 and a 0.162 coverage advantage over STACCA while preserving full UAV connectivity.
By Jiahao Huang, Rongpeng Li, Zhifeng Zhao, Guoru Ding, Honggang Zhang
arXiv:2608. 17841v1 Announce Type: cross Abstract: Multi-armed bandit algorithms are evaluated by regret, yet comparable regret can coexist with different allocations across independent runs.
By Kaifei Wang, Yinyu Ye, Han Zhong
The paper introduces Task Specialization Fine-Tuning (TSFT), an online framework that allocates a limited fine‑tuning budget across multiple task regions in Contextual Reinforcement Learning. TSFT predicts fine‑tuning performance with a simple parametric model and solves the budget allocation problem exactly using integer linear programming. Experiments on combinatorial optimization, continuous control, and LLM fine‑tuning show that TSFT outperforms baselines in task coverage and approaches oracle performance.
By Jianan Zhou, Jung-Hoon Cho, Tianyue Zhou, Han Zheng, Jie Zhang, Roy Dong, Yining Ma, Cathy Wu