The paper investigates task collapse—a failure mode where online RL fine‑tuning of a pretrained flow‑matching vision‑language‑action policy erodes performance on individual tasks—using a 450M‑parameter SmolVLA policy on LIBERO‑10. Three exploration‑noise strategies are compared: a fixed noise scale, a learned noise network, and an uncertainty‑gated controller that reallocates exploration based on novelty and competence signals without task labels. The uncertainty‑gated controller prevents task collapse across all tested seeds, whereas the other two approaches consistently cause collapse, demonstrating its effectiveness in preserving task performance during fine‑tuning.
By Mehmet Turan Yard{\i}mc{\i}, Yunus Emre \c{C}o\u{g}urcu
arXiv:2609. 29881v1 Announce Type: cross Abstract: Choosing Shellsort gaps is a well-known open problem.
By Bo Liu
The paper introduces Continual Reasoning Gym, a continual‑RLVR environment that sequences text and visual reasoning tasks. It finds that while sequential RLVR shows modest forgetting, its final performance lags behind multitask RLVR (MTRL) because forgetting explains only part of the gap. To bridge this, the authors propose Continual Prompt Replay (CPR), which replays previous‑task prompts and regenerates responses with the current policy, achieving on average MTRL‑level performance.
By Lirui Luo, Guoxi Zhang, Hongming Xu, Rongqing Li, Cong Fang, Lifeng Fan
The paper introduces TRACE, an amortized temporal gradient‑inversion attack that reconstructs private observation‑action trajectories from per‑step policy gradients in embodied reinforcement‑learning agents. TRACE exploits cross‑time correlation between gradients and exact action recovery from policy‑head gradients, achieving high reconstruction quality (18.8 dB PSNR) and near‑perfect action recovery with minimal computation. The study demonstrates TRACE’s effectiveness across various neural architectures and input modalities, and suggests that protecting temporal gradient streams may require sequence‑aware privacy mechanisms.
By Sudip Bhujel, Shanghao Shi, Ruiquan Huang, Ning Zhang, Yang Xiao
The paper introduces DEEPO, a Dual-Entropy Enhanced Policy Optimization method designed to mitigate hallucination in multimodal large language models (MLLMs). It addresses two weaknesses in reinforcement learning: (1) hard queries with high semantic entropy produce uniformly wrong samples, erasing advantage signals, and (2) confident-but-wrong tokens become invisible to gradients as the policy sharpens. DEEPO combines semantic‑entropy‑triggered expert prefixes to inject grounded continuations and Renyi preconditioning to counter logit saturation, yielding significant hallucination reduction while maintaining accuracy and training stability.
By Yingxuan Zhuang, Miao Pan, Wangjie Gan, Jingxiao Yang, Fan Wang, Weiming Liu, Cheng Tan, Xuhong Zhang, Jintao Chen
The Pistis Technical Report introduces the Pistis model family, comprising 27B- and 9B-parameter multimodal large language models built on Qwen3.6 and Qwen3.5. The models are developed through a scalable post‑training framework that begins with large‑scale multimodal supervised fine‑tuning and then applies Interleaved Distillation and Reinforcement Learning (IDRL) to integrate on‑policy distillation and reinforcement learning within a single training loop. Two specialized variants—Pistis‑Thinking for deep multimodal reasoning and Pistis‑Agentic for long‑horizon planning, iterative reasoning, and tool use—are produced at both scales, and a system‑level method called Pistis‑Auto‑Harnessing (PAH) further improves inference harness performance without updating model parameters.
By Heyun Chen, Xiaohan Lan, Jiaxi Li, Zhilin Lu, Qi She, Weiwen Xu, Fei Yu, Yujie Zhong, Jinghuan Chen, Zijian Feng, Siyu Jiao, Yiheng Lin, Xinhao Wang, Sihan Yang, Jieyu You, Changbin Zhang, Hengyu Zhang, Xudong Zhang, Yunqing Zhao, Shuai Zheng
The paper introduces TRACER, a multi‑turn user simulator that models evolving user intent and aligns simulated behavior with real interaction trajectories. TRACER is trained first with supervised fine‑tuning on real dialogues and then with reinforcement learning that uses hierarchical outcome‑ and trajectory‑level rewards to address reward sparsity and credit assignment. In real customer‑service sessions, TRACER‑7B outperforms the best baseline by 11.4 conversion F1, achieves the lowest group‑level conversion‑rate error and semantic trajectory distance, and generalizes to out‑of‑distribution scenarios, while human Turing tests show its conversations appear natural. The authors also present the Dynamic Marketing Benchmark, which evaluates both persuasion effectiveness and response quality of large language models through simulated interactions, demonstrating that higher response quality does not always lead to higher conversion rates.
By Geng Chen, Ruotong Pan, Zhirui Yang, Qiqi He, Jiawei Chen, Zhang Yunfei, Chongyuan Chen, Minxuan Lv, Zheng Yang, Win-Bin Huang, Xiangyu Wu, Wenwu Ou
The paper introduces CARE, a contrastive accuracy reward estimation method that adaptively adjusts reasoning length for large language models. By comparing beneficial length adjustments from online sampled responses, CARE applies adaptive length rewards within Group Relative Policy Optimization without extra hyperparameters or inference cost. Experiments on multiple reasoning benchmarks show that CARE improves Pass@1 by up to 4% while reducing reasoning length by 37%, achieving higher token efficiency.
By Zhengdong He, Yunfan Zhou, Jianguo Yao, Haibing Guan, Xijun Li
SciWalker is a framework that automatically synthesizes scientific coding problems by sampling operator chains from scientific library interfaces and using execution feedback to refine generated problem statements, solutions, and tests. It produces 8,178 high‑quality problems across five scientific domains and 32 subdomains, and training a large language model with these problems improves its scientific coding accuracy by nearly 10 percentage points. The approach combines structured workflow composition with verification and quality review to enable scalable, scientifically grounded task generation.
By Chenxi Li, Wenxuan Zeng, Yun Luo, Fangchen Yu, Peng Ye, Yu Cheng, Jun Zhang
The paper introduces Evo-Rec, a three‑stage framework that improves generative recommendation by learning better reasoning traces for Semantic ID (SID) generation. It first aligns SIDs with textual and behavioral contexts, then selects candidate reasoning traces that improve ground‑truth item prediction, and finally refines the reasoning policy via reinforcement learning with catalog‑constrained generation and ranking‑aware feedback. Experiments on Amazon Review datasets show Evo‑Rec consistently outperforms existing discriminative, generative, and reasoning‑enhanced recommenders across all metrics.
By Mengdan Zhu, Yufan Zhao, Sophie Di, Yao Zhao, Tao Di, Yulan Yan, Sridhar Iyer, Liang Zhao
PoEM predicts reinforcement learning outcomes for a new reward function using models already trained on other rewards. If the new reward is a linear combination of existing ones, the new policy’s log-space representation can be expressed as a linear combination of existing log-policies. Even when rewards are not linearly related, log-policies often span a low‑rank subspace, allowing the weighting coefficients to be estimated from reward or basis policy outputs, enabling policy approximation without additional RL training.
By Kimia Hamidieh, Giannis Daras, Antonio Torralba
The paper introduces ICRL4AHT, a large-scale benchmark for evaluating In-Context Reinforcement Learning (ICRL) in Ad-Hoc Teamwork (AHT) scenarios using Overcooked-V2. It provides a diverse teammate suite, a reproducible pipeline, and evaluates history-conditioned ICRL algorithms such as Algorithm Distillation and Decision-Pretrained Transformer. The results show that these methods often perform worse than random baselines and do not improve with longer horizons, underscoring the difficulty of strategic inference under partial observability in AHT.
By Yuheng Jing, Kai Li, Ziwen Zhang, Jiajun Zhang, Zeyao Ma, Jiaxi Yang, Lei Zhang, Zhe Wu, Jinmin He, Junliang Xing, Jian Cheng
The paper critiques current encoded‑prompt safety benchmarks that focus only on harmful requests, showing that such tests can misrepresent a model’s safety. By evaluating the benign arm under the same encoding, the authors reveal a substantial drop in the harm gap—sometimes to zero—indicating that the encoding masks true safety deficiencies. Across multiple large models and training pipelines, they document that the encoding can either hide or falsely inflate safety metrics, and they identify twelve specific instrument defects that contribute to these misleading results.
By Haoyu Zhang, Haowen Xu, Xiao Luo, Hanwen Liu, Yang Chen, Zijian Xiao, Yi Feng, Xiangchen Guan, Mohammad Zandsalimy, Shanu Sushmita
CataOPD introduces a new framework for improving large language model reasoning by combining reinforcement learning and on‑policy distillation. The method treats the teacher as a catalyst that expands the student’s reachability, using Self‑Rescue Routing to find correct trajectories through additional on‑policy sampling and Catalytic‑Guided Self‑Resolution to elicit verified student trajectories. Barrier‑Weighted Internalization further focuses updates on decisive tokens, leading to better performance on unseen problems and improved out‑of‑distribution generalization.
By Wenjin Liu, Chenxi Wang, Jiapu Wang, Zhe Cui, Anh Tuan Luu, Haoran Luo
The paper revisits realizable multiclass PAC learning with bandit feedback, correcting a previously claimed lower bound on sample complexity. It introduces a new anchored dimension, “aBDS,” and establishes a constant‑free three‑part lower bound, while also providing tighter upper bounds that eliminate dependence on the total label count. The authors demonstrate that the optimal sample complexity can vary dramatically even among classes with identical dimensional profiles, revealing a confidence direct‑sum phenomenon and a rank‑saturation phase transition.
By Guangjian Zhang
The paper presents a graph-based framework for large-scale railway network management, combining a hierarchical Bayesian model with a Gaussian Process on a graph kernel to infer spatially correlated maintenance environments from Swiss Federal Railways data. It introduces a topology-aware Multi-Agent Reinforcement Learning system that uses graph neural networks and Transformers to optimize network-level policies. The approach demonstrates scalability via zero-shot transfer learning, enabling agents trained on small network segments to perform effectively on unseen large networks, outperforming heuristics and standard MARL baselines while reducing training time.
By Giacomo Arcieri, Gregory Duth\'e, Christophe Muller, Konstantinos G. Papakonstantinou, Daniel Straub, Eleni Chatzi
EAGER is a reinforcement learning framework for generative event extraction that uses fine‑grained verifiable rewards and Schema‑Contrastive Advantage Estimation to address advantage collapse in sparse binary reward settings. The reward design focuses on structural validity, extraction accuracy, groundedness, coverage, over‑generation, and span precision. Across seven benchmark datasets, EAGER outperforms prompting, supervised fine‑tuning, and previous reinforcement learning baselines, achieving a substantial improvement over the strongest prior method.
By Omar Adjali, Siting Liang, Omair Shahzad Bhatti, Daniel Sonntag
The paper "Learning to Ideate for Scientific Impact" explores using delayed signals of scientific uptake—specifically citation-normalized impact—as feedback to steer large language models toward generating high‑impact research ideas. The authors build a dataset of over 100,000 computer science papers, train a reward model to predict citation impact from goal‑idea pairs, and align an idea generator via supervised fine‑tuning and reinforcement learning. Evaluation with a reference‑grounded protocol shows that the RL‑tuned model consistently produces ideas with higher estimated impact than baseline models.
By Shubham Kale, Aniketh Garikaparthi, Manasi Patwardhan
IterSynth introduces a role-decoupled, iterative synthesis framework for deep search agents, separating planning and synthesis into distinct Planner and Synthesizer modules that maintain a persistent summary state. This design mitigates role coupling and context noise, while the new Role-Decoupled Policy Optimization (RDPO) enhances training by combining outcome rewards with turn-level rubric evaluations. Experiments on five long-horizon benchmarks show IterSynth-8B outperforming prior ≤8B agents by 4.2% and delivering significant zero-shot gains over ReAct on proprietary models.
By Xingyu Wu, Yuchen Yan, Zhengxi Lu, Siqi Chen, Xin ZHANG, Aiting Liu, Chao Deng, Jie Liu, Jin Ma, Jian Shao, Jun Xiao, Yongliang Shen
The paper introduces a neuro‑symbolic architecture that combines symbolic planning, reinforcement learning, and a large language model (LLM) to address novelties in dynamic open‑world environments. The LLM is used to identify missing operators, generate symbolic plans, and write reward functions, enabling the reinforcement learning agent to learn control policies for newly identified operators. The proposed method outperforms state‑of‑the‑art approaches in both operator discovery and learning within continuous robotic domains.
By Hong Lu, Pierrick Lorang, Timothy R. Duggan, Jivko Sinapov, Matthias Scheutz