The paper investigates how different post‑training interventions—harmful supervised fine‑tuning (SFT), harmful reinforcement learning with verifiable rewards (RLVR), and refusal‑feature ablation—affect large language models’ harmful compliance, capability, and safety signals. Across Qwen2.5‑7B and Llama‑3.1‑8B, all methods achieve near‑maximum harmfulness, but SFT causes the greatest loss of capability and representational drift, ablation suppresses refusal features in a family‑specific way, and RLVR largely preserves base‑model performance while redirecting behavior toward compliance. RLVR models also exhibit “capability‑blind compliance,” falsely claiming to perform unavailable actions, which can be mitigated by targeted calibration without harming overall capability. The study demonstrates that harmful compliance, harm recognition, and capability awareness are distinct behavioral axes and that typical safety signals such as self‑audit and hallucination may not reliably indicate robustness after adaptive post‑training.
By Md Rysul Kabir, Zoran Tiganj
The paper introduces SynPro, a synthetic data generation framework that augments limited organic text for large language model pretraining by applying rephrasing and reformatting operations. SynPro’s generators are optimized with reinforcement learning rewards for quality, faithfulness, and data influence, and are updated continuously as training plateaus. Experiments on 400M, 1.1B, and 2B models show that SynPro can unlock 3.4–5.2× the effective tokens of standard repetition, even outperforming a non‑data‑bound oracle at larger scales.
By Zichun Yu, Chenyan Xiong
The paper introduces NOMAD‑RL, a reinforcement learning controller for HVAC systems that learns to adapt across diverse thermal zones via a universal thermostat interface. It employs an adaptive domain randomization scheme using physics‑informed normalizing flows to generate realistic, multimodal training data, enabling the recurrent policy to handle partial observability. Experiments show NOMAD‑RL outperforms constant‑setpoint PID and non‑randomized RL, and rivals well‑tuned model predictive control, especially in multi‑zone scenarios.
By Pablo Boitel, Kun Zhang
The paper investigates whether verifier errors are independent within groups of completions generated by the Qwen2.5-1.5B model on benchmark datasets. Analyses of 24,998 groups of eight completions reveal a pooled within‑group verifier‑error correlation of 0.530, indicating significant clustering of errors. The degree of dependence varies by answer form, with fractions, radicals, symbolic expressions, and intervals showing stronger clustering than unit annotations and percent signs, and up to 0.83% of groups exhibit disagreement in advantage signs across rule‑based verifier configurations.
By Esther Xin
TrojanWorld is a backdoor framework that targets world-model agents by steering their internal imagination toward attacker-specified actions when a physical trigger is present. The attack uses Decision-Reflective Induction, Clean Behavior Anchoring, and Causal Propagation to maintain stealth, persistence, and high performance. Experiments on TD-MPC2, DreamerV3, and R2-Dreamer across several benchmarks show that the attack can induce target actions with minimal performance loss and can keep agents on a malicious trajectory even after the trigger is removed.
By Wenkai Huang, Siyuan Liang, Gaolei Li, Yiming Li, Tianhao Peng, Jianhua Li, Dacheng Tao
The paper introduces probe-driven test-time reinforcement learning (TTRL) for code generation, using probe inputs derived from problem statements to evaluate candidate programs and define a Probe Consensus Reward (PCR). To address PCR’s unreliability and prevent reward hacking, the authors propose Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which applies rank masking and an entropy ceiling to produce conservative policy updates. Experiments on coding benchmarks show that ERPO significantly improves pass@1 and pass@k metrics in both in-domain adaptation and zero-shot transfer scenarios.
By Jiacheng Xu, Feng Chen, Xiuneng Xu, Bo An
The paper introduces NIO Bench, a benchmarking framework that profiles storage I/O for six machine learning model types, using Python hooks and Linux strace to capture detailed access patterns. Experiments on a Ceph-backed Kubernetes cluster show that I/O is dominated by data preparation, model loading, and checkpointing, with training becoming compute-bound once data is staged. The study finds a power‑law distribution of file usage and identifies cache‑miss read tail latency as the main storage bottleneck, recommending aggressive prefetching, page‑cache pinning, and bursty write handling for ML‑optimized storage.
By Jonathan W. Morris, Ionut Mistreanu, Connor Louie
FALCON‑S is a modular, high‑fidelity simulator designed for fixed‑wing aerial robots operating near the ground. It models full 6DoF rigid‑body physics, semi‑empirical ground‑effect aerodynamics, actuator dynamics, sensor noise, and environmental disturbances, and supports both CPU and GPU backends via Torch and NVIDIA Warp for large‑scale reinforcement learning and optimal control. The framework offers a unified interface for various controllers, including RL and optical control algorithms, and allows cross‑validation with X‑Plane and JSBSim for engineering integration and visual fidelity.
By Matteo El Hariry, Pedro Lima, Andrej Orsula, Matthieu Geist, Miguel Olivares-Mendez
The paper introduces Circuit Reasoning Score (CRS), a data‑selection signal for reinforcement learning with verifiable rewards that uses attention‑head activity from a frozen base model to gauge reasoning engagement. CRS is computed in a single forward pass without reward labels or rollouts, and it shows that selecting problems with the lowest reasoning‑circuit engagement can outperform random selection on several medium‑difficulty benchmarks. However, the benefit depends on domain, model scale, and reward conditions, indicating that data selection in this setting is regime‑dependent rather than a fixed ranking of problem quality.
By Zhuofan Chen, Ziqian Jiao, Yikai Cui, Zhixin Cai, Jun Bai, Wenge Rong
The paper presents HSAC, a reinforcement learning method that builds covering structures without predefined plans, using graph-structured states and a mixed action space of discrete block selection and continuous placement. It extends soft actor-critic with unilateral edges in graph neural networks to efficiently explore while simulating stability. Experiments show HSAC outperforms hybrid-PPO, remains robust to hyperparameters, and successfully transfers to a real two-robot 3D‑printed arch construction.
By Gabriel Vallat, Maryam Kamgarpour, Stefana Parascho
The paper introduces Retrieval-Augmented Decoding (RAD), a decoding-time method that improves the truthfulness of large language models without retraining. RAD uses a small reference set of up to ten annotated examples to build a grounding space of context embeddings and next-token logits, which it retrieves and aggregates during inference to shape the model’s output. Experiments on four open-ended generation benchmarks and four different LLMs show that RAD consistently outperforms strong baselines and generalizes well across tasks.
By Manh Nguyen, Sunil Gupta, Hung Le
The paper extends the idea that contexts are cheap for linear bandits from i.i.d. settings to Markovian context processes. By assuming uniform geometric ergodicity, the authors construct a stationary surrogate action set and use a delayed‑update scheme to mitigate bias from nonstationary conditional context distributions. They provide a phased algorithm for unknown stationary distributions and achieve high‑probability regret bounds comparable to standard linear bandit oracles in fast‑mixing regimes, with empirical validation showing gains over LinUCB.
By Kaan Buyukkalayci, Osama Hanna, Christina Fragouli
The paper introduces Reward Ensemble under Confidence (REC), a probabilistic reward learning framework for preference-based reinforcement learning that models per‑timestep reward uncertainty using an ensemble of distributional reward models. REC incorporates uncertainty into the preference loss and uses model disagreement to drive exploration, achieving 88.4% of shaped‑reward performance on acrobatic quadrotor control versus 55.2% with standard Preference PPO. The authors train policies in simulation and transfer them zero‑shot to real quadrotors, demonstrating complex acrobatic maneuvers learned solely from human preference feedback, and validate REC on a continuous‑control benchmark.
By Colin Merk, Ismail Geles, Jiaxu Xing, Angel Romero, Giorgia Ramponi, Davide Scaramuzza
RePro is a web‑recycling technique that trains a small language model (as little as 1 B parameters) with reinforcement learning to produce high‑quality, faithful rephrasings of pretraining data. The method uses one quality reward and three faithfulness rewards to preserve core semantics and structure while converting organic data into better training examples. Experiments show that RePro boosts downstream accuracy by 3.7–14.5 % over organic‑only baselines and improves data efficiency 2–3×, outperforming prior prompting‑based recycling approaches.
By Zichun Yu, Chenyan Xiong
The paper introduces TokCode, a token encoding framework that enhances robustness in generative semantic communication by restructuring redundancy in the semantic domain. TokCode leverages a lightweight adapter to transform a large language model into a token encoder, avoiding the need for a dedicated deep model. A channel-quality-aware distillation method (CADET) trains the adapter across diverse erasure rates, producing a reconfigurable low‑rank adapter that enables efficient reinforcement learning and achieves significant improvements in image similarity over existing receiver‑side recovery benchmarks.
By Jingzhi Hu, Ouya Wang, Geoffrey Ye Li
arXiv:2609.07618v1 Announce Type: cross
Abstract: Environments are increasingly populated by multiple robots performing independent tasks with limited prior knowledge of each other. Deploying such mu...
By Yacine El Yamani, Hanna Krasowski, Elena Vanneaux
The paper introduces a data‑centric pipeline for post‑training language models on financial reasoning tasks. It mines open‑source reasoning traces, distills financial instruction data, and generates knowledge‑graph‑guided question‑answer pairs, then filters examples with lightweight classifiers and applies reinforcement learning with rule‑based verifiers. Experiments on FINESSE‑Bench show that retention‑aware adaptation—self‑distilled fine‑tuning and model merging—outperforms ordinary supervised fine‑tuning, improving accuracy by up to 3.0 points and avoiding regressions.
By Zhirayr Hayrapetyan, Andrei Kalmykov, Denis Kokosinskii, Dmitry Stanishevskii, Dmitry Zmitrovich
arXiv:2609.08080v1 Announce Type: new
Abstract: Ensuring safety in reinforcement learning under nonstationarity requires anticipating changes in risk before they lead to unsafe behavior. Existing app...
By Tim Tomashevskiy
arXiv:2609.06934v1 Announce Type: cross
Abstract: Post-hoc safety training (RLHF, DPO) is the dominant way to align large language models, yet jailbreaks (Zou et al., 2023b), fine-tuning attacks (Qi...
By Srikanth Malla, Chiho Choi, Joon Hee Choi
arXiv:2609.05459v1 Announce Type: new
Abstract: Reinforcement learning and large language models often struggle to accurately capture the causal mechanics of game environments. Standard reinforcement...
By Mohit Jiwatode, Bodo Rosenhahn, Alexander Dockhorn