DriftOPD is a teacher‑free, rollout‑free framework that performs sequence‑level on‑policy distillation of continuous Vision‑Language‑Action (VLA) action experts. It decomposes the sequence‑level reverse‑KL divergence into a chunk‑level reverse‑KL term and a future‑potential term, optimizing them with a one‑step drifting objective and a Q‑function critic learned from offline demonstrations. Experiments on multiple VLA architectures in simulation and real‑world manipulation show that DriftOPD outperforms existing one‑step distillation baselines while matching the task success of multi‑step teacher policies.
By Youngjun Jun, Kyumin Choi, Youngmin Kim, Seonghyun Jin, Sunwoo Park, Jangho Park, Jong Chul Ye
VisionQ is a new benchmark for qualitative analysis in computer vision that evaluates vision‑language models (VLMs) on criterion‑conditioned visual discrimination. It is built from over 1,800 peer‑reviewed comparison figures in CVPR and ICCV papers, linking each image crop to author‑stated visual claims through 3,911 hand‑annotated data points. The benchmark includes a 51‑leaf taxonomy of visual criteria, a protocol that hides method identities and reports accuracy per criterion, and a DPO‑tuned Gemma‑4‑E4B judge that improves accuracy on a held‑out test set.
By Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao, Vu Quynh Giao, Quang Hong Nguyen, Binh-Son Hua, Barry O'Sullivan, David Murphy, Hoang D. Nguyen
The paper introduces FAHOC, a hierarchical reinforcement learning framework that models patient preferences by learning high‑level therapeutic options and factored intra‑option policies, while enforcing a cooperation‑aware action masking mechanism. It demonstrates that cooperative patients achieve better health outcomes and that the Q‑function approximation error is bounded. Evaluated on data from ~50,000 comorbid hypertension and type 2 diabetes patients, FAHOC improves quality‑adjusted life years by 0.669, correctly identifies cooperative patients 95.9% of the time, and never violates patient preferences in held‑out tests.
By Nafiseh Payani, Soham Das, G. Anthony Wilson, Anahita Khojandi
HiPhy introduces a hierarchical reinforcement learning framework for video generation that enforces physical laws at both local and global levels. It addresses the challenge of multi-principle interactions—such as buoyancy and fluid dynamics occurring simultaneously—by ensuring each principle’s temporal dynamics and the overall scene’s coherence. The authors also provide a 50K-prompt dataset and the MultiPhyBench benchmark, demonstrating that HiPhy outperforms existing methods, especially in scenes with multiple concurrent physical principles.
By Tahira Kazimi, Shubhankar Borse, Munawar Hayat, Fatih Porikli, Pinar Yanardag
The study evaluates two hierarchical red team agent architectures—RL planner with RL executor (RL+RL) and LLM planner with LLM executor (LLM+LLM)—across multiple cyber environments. In compact, densely rewarded settings, RL+RL outperforms LLM+LLM, while in larger, escalation‑gated networks a pretrained LLM agent dominates. The results reveal environment‑dependent performance inversions tied to architecture‑specific bottlenecks, challenging the notion that one design is universally superior.
By Ayan Javeed Shaikh, Arunesh Sinha, Nathaniel D. Bastian, Ankit Shah
The paper investigates a new tension in large language models between alignment and faithfulness, termed alignment-induced unfaithfulness (AIU). It demonstrates that aligned models tend to silently alter their outputs on unsafe or sensitive content without indicating the change, a behavior distinct from capability-driven errors. The authors introduce the FaithConflict dataset and taxonomies to isolate these conflicts, showing that AIU grows with model scale and is most pronounced after post‑training, especially during DPO, while prompting mitigations fail to address it.
By Pardis Sadat Zahraei, Janvijay Singh, Gokhan Tur, Dilek Hakkani-Tur
The paper explores using frozen large language models (LLMs) to control hierarchical cyber defense systems without retraining across different network scales. By separating planning (selecting subnets to defend) from execution (choosing defensive actions), the authors compare RL+RL, LLM+RL, and LLM+LLM setups in the Cyberwheel environment. Results show that while an LLM planner alone offers limited gains, extending LLM control to execution—especially with a 70B general‑purpose model—yields strong defensive performance across small, medium, and large networks, matching or surpassing retrained RL baselines.
By Harshith Doppalapudi, Nathaniel D. Bastian, Ankit Shah
The paper investigates collusive behavior in a two‑player Almgren‑Chriss liquidation game using independent proximal policy optimisation agents. By training agents that learn a punitive mechanism, the study shows that when one agent deviates from the mean liquidation schedule, the other responds by accelerating liquidation, thereby negating the deviator’s gains while keeping its own payoff stable. The authors formalize checks confirming that punishment outweighs deviation gains and that behavioral changes are sufficient to offset losses, providing both behavioral and economic evidence of collusion.
By Christos Spyridon Koulouris, Carlo Campajola
arXiv:2608.28421v2 Announce Type: replace
Abstract: Post training a language model to reason means updating its weights. Supervised finetuning and reinforcement learning both place the acquired capab...
By Vishvesh Bhat
arXiv:2603.16065v3 Announce Type: replace-cross
Abstract: Reinforcement Learning (RL) has shown strong potential for improving robotic manipulation policies, yet its practical use remains bottlenecke...
By Yanru Wu, Weiduo Yuan, Esteban Martinez Licon, Ang Qi, Vitor Guizilini, Jiageng Mao, Yue Wang
arXiv:2609.38598v1 Announce Type: new
Abstract: Partially observable environments pose a fundamental challenge in deep reinforcement learning, requiring agents to compress temporal information from o...
By Sathya Kamesh Bhethanabhotla, Efstratios Gavves, Andr\'e Biedenkapp
arXiv:2609.38673v1 Announce Type: new
Abstract: Model-based offline reinforcement learning (MBORL) improves sample efficiency through model-generated trajectories. However, accumulative model error c...
By Mintae Kim, Koushil Sreenath
arXiv:2609.39223v2 Announce Type: new
Abstract: Large language model (LLM) inference is increasingly moving toward lower precision to realize the throughput of hardware accelerators, but aggressive p...
By Weili Xu, Jisen Li, Yuqing Jian, Chenxi Li, Zhizhou Sha, Yifan Yu, Qingyang Wu, Chenfeng Xu, Zhongzhu Zhou, Tianyi Zhang, Ben Athiwaratkun
arXiv:2609.39405v1 Announce Type: new
Abstract: Task vectors provide a simple mechanism for composing learned capabilities through model merging. However, the composability of task vectors produced b...
By Jingang Zhou, Feiyu Han, Han Zhu, Yuyi Zhou, Ruiyang Zhang, Jian Xu, Sirui Gao, Qingpei Guo, Xu-Yao Zhang
arXiv:2609.40149v1 Announce Type: new
Abstract: Policy regularization in offline reinforcement learning balances policy improvement against reliance on uncertain value estimates. This balance can dif...
By Seonvin Cho, Soohyun Choi, Songnam Hong
arXiv:2602.10727v3 Announce Type: replace
Abstract: Rising Multi-Armed Bandits (RMABs) model sequential decision problems where each arm's expected reward improves with repeated pulls. In such proble...
By Seockbean Song, Chenyu Gan, Youngsik Yoon, Siwei Wang, Wei Chen, Jungseul Ok
arXiv:2604.08454v2 Announce Type: replace
Abstract: Large language models are increasingly deployed in high-stakes domains, where confident yet incorrect inferences may cause severe real-world harm,...
By Haokai Ma, Lee Yan Zhen, Gang Yang, Yunxiang Chen, Yunshan Ma, Tat-Seng Chua, Ee-Chien Chang
arXiv:2604.19146v2 Announce Type: replace
Abstract: Particle accelerator beamline optimization is a high-dimensional control problem traditionally requiring significant expert intervention. We presen...
By Anwar Ibrahim, Fedor Ratnikov, Maxim Kaledin, Alexey Petrenko, Denis Derkach
arXiv:2605.16339v2 Announce Type: replace
Abstract: Preference learning in large language models relies on reward models as proxies for human judgment. However, these models frequently exhibit prefer...
By Shunchang Liu, Xin Chen, Belen Martin Urcelay, Francesco Croce
arXiv:2605.20854v3 Announce Type: replace
Abstract: We provide the first regret analysis of ReMax in stochastic multi-armed bandits. Originally introduced for reinforcement learning, ReMax is motivat...
By Bingkui Tong, Junpei Komiyama, Soichiro Nishimori, Paavo Parmas