The paper introduces GPlan, a generative framework for Generative Spatiotemporal Intent Sequence Recommendation (GSISR) that integrates large language model reasoning into lightweight models. It employs Progressive Implicit CoT Distillation to compress reasoning into latent tokens for low-latency inference, and Spatiotemporal Counterfactual DPO to align plans with real-world constraints, improving sequence coherence and context responsiveness. Experiments and online A/B tests confirm the effectiveness of these techniques, and the authors provide an implementation and dataset on GitHub.
By Sicong Wang, Ruiting Dong, Yue Liu, Bowen Zheng, Jun Meng, Jie Li, Shuaijun Guo, Yu Gu, Fanyi Di, Xin Li
The paper investigates how to efficiently adapt a pretrained Q‑function from offline data to a new environment using limited online interactions. It establishes a minimax lower bound showing that adaptation can be as hard as starting from scratch when the pretrained function is only slightly suboptimal. The authors then introduce O2O‑LSVI, an algorithm that, under a new structural assumption on the pretrained values, achieves better sample complexity than pure online reinforcement learning, and validate its effectiveness with neural‑network experiments.
By Shangzhe Li, Weitong Zhang
The paper introduces Residual Advantage (RA), a new guidance method for reinforcement learning with verifiable rewards (RLVR) that treats the teacher–student probability residual as a bounded one‑step reward, subtracts the student’s state value to form an advantage, and centers it within each response. RA preserves the verifier’s mean label while redistributing credit among response steps, and its extension CoRA updates the teacher model using verifier advantages to adapt guidance to the student’s attempts. Experiments with Qwen3 models show that RA combined with GRPO or REINFORCE++ consistently outperforms teacher‑only on‑policy distillation across three mathematical benchmarks, improving macro Avg@8 and Pass@8 scores, with CoRA providing additional gains.
By Xiaobing Chen, Zhiqi Pang
HarnessSQL introduces a harness‑native post‑training framework that keeps the full stateful, multi‑turn interaction structure of real‑world database agents during both supervised fine‑tuning and reinforcement learning. By building isolated executable database environments with hidden execution oracles and retaining only verified trajectories for full‑sequence SFT, the method enables execution‑reward RL that aligns training with inference. Experiments on Spider 2.0‑SQLite show dramatic gains in execution accuracy for compact models, and the approach transfers effectively to out‑of‑distribution interactive benchmarks such as BIRD‑Interact and LiveSQLBench.
By Haolin Yang, Jipeng Zhang, Jian Xie, Shuaishuai Gong, Sirui Han, Yike Guo
The paper introduces the AutoRegressive Retriever (ARR), a multimodal retrieval model that iteratively refines a query embedding by selecting informative items and incorporating their content. ARR alternates between retrieval and query update, using stepwise contrastive supervision and reinforcement learning to optimize item selection and final ranking. Experiments show ARR outperforms baselines on in‑domain and zero‑shot benchmarks, with feedback improving both inference performance and initial query embeddings.
By Jianfei Zhao, Yifan Wang, Feng Zhang, Xin Sun, Chong Feng, Zhixing Tan, Yang Luo, Boyuan Pan, Xu Kai, Yao Hu
The paper proposes a method for allocating limited counterfactual annotation resources from multiple sources—such as domain experts and large language models—to improve off-policy evaluation in contextual bandits. By formulating an integer optimization problem that balances source-specific costs and error rates, the authors derive a majorization‑minimization algorithm that iteratively reduces estimator variance. Experiments on synthetic clinical and education bandit data demonstrate that the optimized allocation cuts mean‑squared error by over 20% compared to no annotation.
By Biao Xiang, Ali Eshragh, Yuexing Li, Kai Wang
InfiFPO is a new preference optimization technique for implicit model fusion that replaces the reference model in Direct Preference Optimization with a fused source model, preserving probability information and avoiding complex vocabulary alignment. By incorporating probability clipping and max‑margin fusion, it aligns a pivot model with human preferences while distilling knowledge from multiple source models. Experiments on 11 benchmarks show that InfiFPO consistently outperforms existing fusion and preference optimization methods, raising Phi‑4’s average score from 79.95 to 83.33 and improving performance in mathematics, coding, and reasoning tasks.
By Yanggan Gu, Yuanyi Wang, Zhaoyi Yan, Yiming Zhang, Qi Zhou, Fei Wu, Hongxia Yang
SP-DocReader introduces a self‑play framework for OCR that focuses on residual errors after supervised fine‑tuning. It uses Reading Discrepancy Masking to align reference and generated tokens via a longest common subsequence and scores unmatched positions with full conditioning prefixes. The method adds a Focused Fidelity Loss for direct negative log‑likelihood supervision at unmatched ground‑truth positions, training only the OCR module while keeping the backbone frozen, and achieves significant character error rate reductions and ANLS improvements on multiple backbones.
By Wenjie Liao, Xiaohui Song, Liangjie Zhao, Haonan Lu
The paper introduces Q-Shaped Options (QSO), a hierarchical reinforcement learning framework that learns action abstractions between consecutive levels using shared encoders shaped by Q functions. QSO employs distinct state-value functions, Q functions, and policies at each hierarchy level, encouraging the low-level Q function to preserve distinctions needed for optimal control while the high-level Q function discards unnecessary distinctions. Experiments on offline goal-conditioned locomotion and manipulation tasks show that QSO learns semantically meaningful option spaces and outperforms baselines, achieving non-zero performance where other algorithms fail.
By Clarisse Wibault, Antoine Gorceix, Antonio L\'eon Villares, Alexey Zakharov, Evangelos Chatzaroulas, Michael Matthews, Eduardo Pignatelli, Jakob Foerster
The paper presents a constrained command‑conditioned Proximal Policy Optimization (PPO) executor for the MicroRTS real‑time strategy environment, enabling a single execution policy to follow diverse strategic commands. A Thompson‑sampling bandit strategist selects command tuples based on in‑game observations of the opponent’s strategy, rather than relying on opponent identity. In controlled experiments, the strategist‑executor system outperforms a flat PPO baseline against three of the four strongest opponents on a training map, including two held‑out opponents, while showing no significant difference against the remaining opponents.
By Nick Leenders, Roy Lindelauf, Joost van Oijen, Boris Cule
The paper introduces Stitching Noisy Diffusion Thoughts, a self‑consistency framework that transforms low‑cost diffusion‑sampled reasoning into a reusable pool of step‑level candidates. By sampling many diverse trajectories with a masked diffusion language model, scoring each intermediate step with a process reward model, and stitching the highest‑quality steps across trajectories, the method recomputes only the final answer, separating exploration from evaluation. Experiments on math reasoning benchmarks show that step‑level recombination improves accuracy by up to 23.8% on six tasks while reducing latency by up to 1.8× compared to traditional diffusion models and unified architectures.
By Roy Miles, Aysim Toker, Andreea-Maria Oncescu, Jiankang Deng, Ismail Elezi
The paper introduces RAG‑Stress, a diagnostic protocol that tests how retrieval‑augmented generation models handle misleading evidence. By fixing the question and reference answer, editing a single assertion to support an incorrect answer, and varying source‑priority policies and evidence positions, the study measures misleading rate (MR) and clean accuracy across fifteen systems on multiple QA datasets. Results show that prioritizing documents increases MR by 10.9–13.5 percentage points, with the highest MR when the misleading span appears at the end of the evidence.
By Shunyuan Zhou, Hao Chen, Tianyu Wang, Goose Lin, Zaiyuan Wang, Haiying Zhao
The paper introduces GRPODropout, a technique for online reinforcement learning rollouts that selectively removes a small number of high‑probability positive‑advantage rollouts before policy updates. By re‑centering the retained advantages, the method reduces entropy collapse and improves exploration without adding significant computational cost. Experiments demonstrate higher accuracy and actor entropy than the baseline GRPO while using fewer rollout samples.
By Hexuan Deng, Zihao Yan, Xuebo Liu, Shuo Nie, Yue Wang, Chen Wang, Zhaohua Zhang, Tianwen Jiang, Qiuyong Xiao, Jihong Zhang, Min Zhang
MIMESIS is a user simulator designed to train interactive language agents by mimicking real user behavior. It is trained on human conversations with explicit reasoning supervision and incorporates 13 realistic behavioral patterns. Experiments show that agents trained with MIMESIS outperform those trained with GPT-5.5 across multiple unseen simulators, and the proposed Coached On-Policy Self-Distillation further improves agent performance by providing dense, token-level supervision.
By Hoang Phan, Dat Huynh, Andrey Zhmoginov, Qi Zeng, Wancen Mu, Yue Cao, Shengjie Bi, Yun He, Changdae Oh, Deren Lei
The paper introduces Trust Region Q Adjoint Matching (TRQAM), an off‑policy fine‑tuning algorithm that stabilizes reinforcement learning of pretrained flow policies by adaptively controlling the path‑space KL divergence between the fine‑tuned and pretrained policies. TRQAM achieves this through projected dual descent, which adjusts a trust‑region parameter λ during sampling and ensures the KL penalty is exactly weighted in the stochastic optimal control objective. Experiments on 50 OGBench tasks show TRQAM consistently outperforms prior methods, achieving a 68% success rate in offline RL versus 46% for the strongest baseline.
By Yonghoon Dong, Kyungmin Lee, Changyeon Kim, Jaehyuk Kim, Jinwoo Shin
The paper investigates whether large language models (LLMs) can learn from rewards presented in context, a process termed in‑context reinforcement learning (ICRL). Through experiments on four benchmarks and six models, the authors find that while LLMs read reward signals, altering or removing these signals has little effect on performance, and that trajectory structure—rather than reward content—drives improvement. These results suggest that direct ICRL behaves more like standard in‑context learning (ICL) than true inference‑time reinforcement learning.
By Minchan Kwon, Seunghee Koh, Sunghyun Baek, Minsung Bae, Junmo Kim
The paper introduces Evolving Functional REpertoires (EFRE), a method that replaces a single prompt with a dynamic set of functions to enable large language models to learn new tasks sequentially without forgetting. EFRE refines existing functions when updates are compatible and creates new ones when updates conflict, thereby mitigating catastrophic forgetting and overfitting. Experiments on a three‑task continual‑learning stream show EFRE outperforms GRPO by 7.50 percentage points and reduces performance drop on a subsequent task by only 1.56 percentage points compared to base prompt optimization.
By Fengyuan Liu, Yue Wang, Hangxi Guo, Fengyuan Liu, Chenxu Wu, Yanguang Liu, Mengnan Du
The paper introduces a method for risk‑averse machine learning that uses credal sets—collections of probability distributions—to capture epistemic uncertainty about the true loss distribution. By producing predictions as credal sets and applying a novel decision rule that selects a single predictive distribution for conditional value‑at‑risk (CVaR) minimization, the approach aims to reliably avoid catastrophic decisions across classification, distribution shift, and reinforcement learning scenarios while maintaining near‑optimal expected performance.
By Timo L\"ohr, Paul Hofman, Maximilian Muschalik, Eyke H\"ullermeier
UXBench Pro is a new benchmark that evaluates personalized user experience in multi-turn dialogue interactions. It contains 1,000 test instances from real user interactions across 12 task scenarios and 82 domains, each paired with a FACTORS user profile describing seven behavioral facets. The benchmark introduces a dual-perspective evaluation: a personalized User Reward Model for third‑person judgment and Sim4Eval, a user simulator for first‑person evaluation across four cognitive state dimensions, along with meta‑benchmarks URMBench and USimBench to assess evaluator reliability.
By Mengze Hong, Zeyang Lei, Wenbo Shang, Xia Zeng, Xiying Zhao, Qi Zhu, Chen Jason Zhang, Di Jiang, Taiming Fu, Qiongyi Zhou, Qinghe Chang, Fubao Zhang, Chenxuan Ma, Minlong Peng, Jinfeng Huang, Zineng Zhou, Jindou Wu, Muge Qi, Sijun He, Xin Cui, Di Liang, Yuan Hua, Davey Chen
NavGPT-3 is a hierarchical navigation runtime that links a large language model trained with long‑horizon reinforcement learning to a low‑latency action policy, enabling an embodied agent to reason, act, and monitor in parallel threads. The system’s OS‑like scheduler selects which thread controls the robot, allowing rapid interruption and reaction to real‑world events. With the NavGPT VLA action policy, the complete harness achieves state‑of‑the‑art performance on R2R‑CE and matches human performance on RxR‑CE, demonstrating a significant advance in autonomous navigation.
By Gengze Zhou, Yicong Hong, Jiazhao Zhang, Xunyi Zhao, Jian Zhou, Zixing Lei, Zun Wang, Chongyang Zhao, Xionghui Chen, Stephen Gould, Anton van den Hengel, Qi Wu