The paper investigates how reinforcement learning agents trained in one cyber simulation can be transferred to other simulators or real environments. It introduces a framework that decouples state alignment from action translation, allowing zero‑shot policy transfer without retraining. Experiments across four cyber platforms show that transferred policies can preserve performance in closely aligned settings and achieve substantial win rates in more divergent environments.
By Sabrina Saika, Yinuo Du, Aritran Piplai
arXiv:2505. 03155v2 Announce Type: replace Abstract: Policy gradient (PG) methods have played an essential role in the empirical successes of reinforcement learning.
By Max Qiushi Lin, Jincheng Mei, Matin Aghaei, Michael Lu, Bo Dai, Alekh Agarwal, Dale Schuurmans, Csaba Szepesvari, Sharan Vaswani
The paper introduces temporal self‑distillation for reinforcement learning with verifiable rewards (RLVR), proposing that a policy can learn from a stronger future checkpoint of itself. Two methods—Near‑Future Policy Optimization (NPO) and Near‑Future Policy Distillation (NPD)—use verified future‑self trajectories and token‑level transfer, respectively, while AutoNPO adaptively selects the optimal future checkpoint. Experiments on eight image‑text benchmarks show that near‑future teachers yield higher performance than far‑future ones, indicating that the balance between new capability and learner compatibility is key.
By Chuanyu Qin, Chenxu Yang, Qingyi Si, Naibin Gu, Dingyu Yao, Zheng Lin, Peng Fu, Nan Duan, Jiaqi Wang
The paper proposes a multi‑objective framework that integrates reinforcement learning‑based multiple instance learning (RL‑MIL), adversarial debiasing, and preference‑conditioned hypernetworks to predict student performance while addressing fairness concerns. Each student is represented as a bag of weakly labeled interactions, and an RL agent selects informative instances for classification. Although the baseline RL‑MIL achieves strong accuracy, the hypernetwork extensions fail to produce a smooth trade‑off between predictive performance and Equalized Odds due to mode collapse, objective dominance, and weak gradient propagation, indicating that additional mechanisms are needed for stable, fair RL‑MIL systems.
By Bente Hinkenhuis, Seyed Sahand Mohammadi Ziabari, Ali Mohammed Mansoor Alsahag
AURAL is a speech language model that performs adaptive latent reasoning by modeling multiple plausible reasoning continuations in latent space and jointly predicting chunks of future states, thereby reducing sequential forward passes and latency. The authors introduce a large bilingual dataset, AuralReason-683K, containing concise chain‑of‑thought annotations for emotion recognition, empathetic dialogue, and general reasoning, and use reinforcement learning (AURAL‑RL) to reward concise, high‑quality reasoning that adapts to problem difficulty. Experiments on two backbones show that AURAL‑RL matches or exceeds chain‑of‑thought reinforcement learning while achieving significant latency reductions, such as an 11.8× speed‑up on Qwen2.5‑Omni.
"whyItMatters":"The work demonstrates that latent reasoning can match the performance of explicit chain‑of‑thought methods while dramatically cutting response time, addressing the trade‑off between intelligence and speed in speech language models."
By Yuxiang Wang, Kunyu Feng, Yuancheng Wang, Zihang Liu, Shengbo Cai, Qinke Ni, Wan Lin, Tao Feng, Yingda shen, Ming-Hao Hsu, Zhixian Zhao, Liqiang Zhang, Teddy Sun, Steve Yves, Zhizheng Wu
The paper introduces ReQRL, a method for learning quasimetric geometry in goal-conditioned reinforcement learning by constraining the critic’s value gradients with finite-horizon reachability. It decouples dynamical reachability from boundary geometry, estimating both from data using state-constrained optimal control principles. Experiments on OGBench show that ReQRL matches or surpasses existing quasimetric and offline GCRL approaches.
By Daisuke Yamada, Travis Pence, Vikas Singh
The paper introduces Function-Structured Graph Reinforcement Learning (FSG‑RL), a framework that links subproblem graphs to Python code and uses multiple verifiers for feedback. It first trains a policy via supervised fine‑tuning to generate code from function graphs, then refines it with Group Relative Policy Optimization (GRPO) that employs answer‑gated rewards and span‑level credit assignment. On a benchmark combining GSM8K, MathQA, MATH, and Omni‑MATH, GRPO raises final‑answer accuracy from 43.25 % to 67.50 % and full‑solution success from 32.25 % to 52.25 %, with further improvements when teacher supervision is added.
By Zihan Liu, Xurong Xie
The paper introduces Reachability Analysis-Informed Reinforcement Learning (RARL) for designing deterministic multi‑impulse interplanetary transfers. RARL uses local first‑order reachability maps to bound velocity perturbations and selects intermediate waypoints, which are then translated into maneuvers via Lambert reconstruction and a terminal two‑impulse solution. Numerical experiments on an Earth‑Mars benchmark show that RARL achieves a mean maneuver cost only 1.72% above a validated convex programming reference and can be trained once to handle a wide range of departure states, achieving 100% feasibility on 10,000 held‑out Monte Carlo departures.
By Yashdeep Chaudhary, Roberto Armellin, Harry Holt
The paper studies how an eavesdropper can adaptively attack quantum key distribution (QKD) systems when channel noise and device drift vary over time. By modeling the attack as a constrained Markov decision process and using reinforcement learning to jointly search gate structures and rotation angles, the authors construct compact attack circuits that perform near the theoretical upper bound for both device‑independent E91 and BB84 protocols under realistic noise models. The results show that adaptive attacks can significantly increase the eavesdropper’s information compared to fixed‑circuit strategies, and that the learned attacks recover known optimal cloners and key‑rate bounds.
By Marcel Mordarski, Benjamin Gras, Abdelrahman Shehata, Daniel Budina, Roberto Bondesan
The paper investigates why Group Relative Policy Optimization (GRPO) benefits from per‑prompt normalization by examining the local curvature of the sequence‑level policy gradient. It shows that standard deviation normalization acts as an adaptive gradient, yielding a provably faster convergence rate than unnormalized REINFORCE under mild conditions, with the improvement tied to the average within‑prompt reward standard deviation. The authors also propose IS‑GRPO, an importance‑sampling variant that maintains alignment with the full gradient and offers a tighter convergence guarantee, and empirically validate these theoretical insights on GSM8K and MATH datasets at 1.5B and 7B model scales.
By Cheng Ge, Caitlyn Heqi Yin, Hao Liang, Jiawei Zhang
The paper introduces a framework for optimal policy improvement in reinforcement learning, defining it as the best single update under given constraints. It shows that restricting improvement to a subset of states is equivalent to solving an induced Markov Decision Process, linking planning with explicit or implicit models to optimal policy improvement. The authors develop a novel operator for greedification under approximate evaluation, demonstrating empirical gains across several RL algorithms and settings.
By Yaniv Oren, Viliam Vadocz, Wiktor Zabka, Thomas Evers, Jan Robine, Wendelin B\"ohmer, Matthijs T. J. Spaan, Martha White, Hendrik Baier, Fenghui Yu
The paper introduces an agentic forecasting environment built on 2,100+ resolved Polymarket questions, where a language model (Qwen3.5-35B-A3B) learns to gather evidence during rollout via web search, page reading, and financial time series, all filtered to avoid post‑cutoff leaks. Training with single‑epoch GRPO and a Brier‑score reward improves calibration by 30‑40% and reduces search attempts, while the trained policy outperforms four frontier models in evidence‑based forecasting, achieving lower soft‑Brier scores at roughly 5% of the inference cost. The authors release the environment, dataset, and per‑rollout records as a reusable harness for temporal forecasting agents.
By Yusuf Afifi, Artur Kiulian, Anton Polishko, Mykola Khandoga, Hamudi Naanaa, Alina Krasnobrizha
The paper investigates how reducing numerical precision through model quantization impacts the vulnerability of neural networks to model inversion attacks. It provides theoretical bounds on mutual information changes and identifies data-dependent effects, especially at 4‑bit precision. Based on these findings, the authors propose a privacy‑aware post‑training quantization strategy that allocates bits adaptively, calibrates activation ranges, and jointly optimizes weight and activation scaling to improve inversion resistance while preserving model utility.
By Rongke Liu, Youwen Zhu
AdaptManip is a fully autonomous framework that enables humanoid robots to navigate, lift, and deliver objects without human demonstrations. It combines a recurrent state estimator for real‑time object tracking, a whole‑body locomotion policy with residual manipulation control, and a LiDAR‑based global position estimator. Trained entirely in simulation with reinforcement learning, the system achieves zero‑shot deployment on real hardware, outperforming imitation‑learning baselines in adaptability and success rate.
By Morgan Byrd, Donghoon Baek, Kartik Garg, Hyunyoung Jung, Daesol Cho, Maks Sorokin, Robert Wright, Sehoon Ha
The paper introduces entry-state sharpening, a data‑free pre‑training step that prepares a language model’s checkpoint in a sharper, lower‑entropy state before test‑time reinforcement learning (TTRL). By reducing policy entropy, the model can more efficiently use its limited adaptation budget, leading to higher endpoint conversion efficiency across tasks such as MATH, GPQA, and AMC. Experiments with different data‑free objectives (e.g., R‑Zero vs. SPIRAL) demonstrate that the choice of pre‑training objective strongly influences the checkpoint’s readiness for TTRL, and a label‑free self‑distillation intervention can further sharpen the entry state.
By Zhanming Zhang, Vinoth Selvendran
The paper extends mechanistic interpretability of large language models by modeling concepts as low‑dimensional non‑linear manifolds rather than linear subspaces. It introduces a concept‑based alignment (CBA) score to compare these manifolds across layers and models, revealing block structures in intermediate layers, a shift from syntax‑dominated to mixed syntactic‑semantic concepts, and training‑dependent multilingual sharing. The study also shows that alignment patterns differ across model families and training stages, with adjacent stages aligning more closely than distant ones.
By Tido Specht, Elias Benedict Krey, Nils Neukirch, Nils Strodthoff
The paper demonstrates that reinforcement learning with verifiable rewards (RLVR) can improve vision‑language benchmark performance even when models are trained without visual input. When real images are introduced at test time, models trained blind recover about half of the performance gain at 3B parameters and nearly four‑fifths at 7B, but extended real‑image training can erode grounding while benchmark gains persist. The authors propose a visual resolvability rule and show that requiring visual evidence for correct answers leads to significant improvements in target discovery and generalization to unseen question types, while controls confirm that the gains stem from actual visual grounding rather than artifacts.
By Haocun Ye, Xinlong Jiang, Qile Chen, Bingyu Wang, Teng Zhang, Shubai Chen, Tingyu Wu, Zhenkun Zheng, Yiqiang Chen
The paper introduces IDiom, an autoregressive protein language model trained on a large dataset of intrinsically disordered protein regions (IDRs) from AlphaFold, and demonstrates that it can generate sequences matching natural IDR composition, motifs, and disorder. It further presents RL‑SAE, a reinforcement learning approach that uses sparse autoencoder features to steer generation toward specific functional patterns, achieving high activation of targeted features and improved predicted subcellular localization and transcriptional activity. The combination of IDiom and RL‑SAE allows interpretable, composable IDR design by explicitly controlling function‑associated sequence features.
By Jason X. Liu, Sebastian Ibarraran, Frank Hu, Soojung Yang, Xinyu A. Feng, Abigail Park, Anagha Aneesh, Lacramioara Bintu, Alexander R. Dunn, Grant M. Rotskoff
Associate Professor Cathy Wu applies reinforcement learning to design computational tools that map out improvements for complex systems such as transportation. Her work demonstrates how advanced AI techniques can be used to tackle multifaceted societal challenges. By modeling these systems, she aims to identify optimal strategies for enhancing efficiency and performance.
By Michaela Jarvis | MIT Laboratory for Information and Decision Systems
SPIRAL is a reinforcement‑learning framework that trains language models to employ three inference primitives—sequential reasoning within a trace, parallel sampling of independent traces, and aggregation of those traces—within a single compute pipeline. The model first generates multiple independent chain‑of‑thought traces in parallel, then produces a final aggregation trace conditioned on them, with all components optimized end‑to‑end for the reward of the aggregated response. Experiments on reasoning tasks demonstrate that SPIRAL scales efficiently with inference compute, achieving up to 11× better scaling efficiency and 15% higher performance compared to the GRPO baseline when all three primitives are scaled.
By Jubayer Ibn Hamid, Ifdita Hasan Orney, Michael Y. Li, Omar Shaikh, Yoonho Lee, Dorsa Sadigh, Chelsea Finn, Noah Goodman