The paper introduces First-Order In-Context Shapley, an inference‑only method that evaluates data utility for large language model alignment by computing the first‑order term of an in‑context cooperative game over semantic k‑NN neighborhoods in O(k) forward passes. It projects log‑likelihood shifts onto a directed influence graph, yielding three signals—Exemplifying Advantage, Exemplified Advantage, and One‑of‑a‑Kind Score—to identify high‑utility demonstrations and rare prompts. The approach reduces compute by 19× compared to Monte‑Carlo Shapley, filters 1–5% of records, and improves error detection and reward accuracy in DPO training and benchmark auditing.
whyItMatters":"By enabling scalable, inference‑only data auditing, the method addresses label noise and safety inversions that hinder post‑training alignment of large language models, leading to measurable improvements in model reliability and reward accuracy."
By Yunting Song, Matthew Watson, Moustafa Farid Alzantot, Peter Grabowski, Jun Qin
The paper investigates reflective prompt optimization (GEPA) for a multi‑agent deep‑research language‑model, focusing on a topic‑disjoint train/validation/test split to avoid memorization. Across three random seeds, the optimized prompts improve held‑out citation reward from 0.702 to 0.781, with a 95% confidence interval of [0.051, 0.107]. The study shows that the gain is driven by decomposed per‑dimension feedback rather than a single scalar reward, and that the improvement persists even with an out‑of‑family verifier and a hand‑written version of the discovered recipe.
By Elias Lumer, Sahil Sen, Vamse Kumar Subbiah, Kevin Paul
MADS (Multi-Agent Dialog Simulation) is a synthetic data generation pipeline that creates structured, diverse dialogue trajectories, including adversarial conversations that reflect realistic failure scenarios. The resulting MADS-Bench benchmark covers six domains, 55 personas, and 1095 conversations, with about 66% featuring failure-prone situations. Models trained with MADS are evaluated using a two-stage approach combining supervised fine-tuning and reinforcement learning with verifiable rewards to assess robustness.
By Arkajyoti Chakraborty, Adam Zernik, Ishika Kulkarni, Mudit Arora, Hugo Lin, Justin Chiu, Alessandro Di Bari, Neha Gupta, Andreas Stolcke
The paper introduces Trajectory-Aware Policy Optimization (TAPO), a new framework for reinforcement learning that addresses the state distribution shift caused by long-horizon autoregressive rollouts in large language models. TAPO derives a state-distribution correction from the off-policy policy gradient, integrates it into the RL objective, and uses a geometric-mean estimator for variance reduction along with length-aware dynamic clipping to stabilize updates. Experiments on eight competition-level benchmarks show TAPO outperforms existing LLM RL methods, achieving up to 48% improvement on mathematical reasoning and 12% on algorithmic coding tasks while enhancing training stability.
By Zhicheng Cai, Weinan Dai, Yuyan Zhou, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
EmoGavel is a reinforcement learning framework that incorporates verifiable rewards to improve multimodal emotion preference learning. It introduces grounded supervised fine‑tuning (Grounded SFT) to jointly learn preference prediction, error localization, repair, and missing‑evidence recovery, and then applies grounding‑aware group relative policy optimization (GA‑GRPO) to prioritize preference correctness and recover learning signals from all‑wrong groups. Experiments on EmoPrefer and controlled‑error evaluation sets demonstrate that EmoGavel enhances natural preference prediction, controlled‑error discrimination, and maintains strong error‑grounding performance.
By Zilong Huang, Kong Aik Lee, Zezhong Jin, Man-Wai Mak, Tatsuya Kawahara
arXiv:2610. 13653v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces.
By Abiel Malkani Talwar, Christopher Nemeth, Henry Moss
DiffusionART introduces a new reinforcement learning framework for diffusion models that aggregates prompt rewards into a shared direction, eliminating consistency forces inherent in prior methods. It employs a trust‑region‑masked second update for drift‑adaptive restoration, removing the need for EMA tuning and differentiable reward models. Experiments on image and video generation show that DiffusionART trains more efficiently and stably, achieving higher mean rewards with significantly fewer GPU hours compared to existing approaches.
By Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao, Fangyuan Kong, Pengfei Wan, Kun Gai, Yang You
arXiv:2604. 09434v3 Announce Type: replace-cross Abstract: Turbulent boundary layers over aerodynamic surfaces are a major source of aircraft drag, yet their control remains challenging due to multiscale dynamics and spatial variability, particularly under adverse pressure gradients.
By Yuning Wang, Pol Suarez, Mathis Bode, Ricardo Vinuesa
MemReread is a new agentic long‑context reasoning method that uses memory‑guided rereading instead of intermediate retrieval. By decomposing questions and selectively rereading when the final memory is insufficient, it recovers indirect evidence that might otherwise be lost during memory overwriting. A reinforcement‑learning framework further improves length extrapolation and adaptively controls the number of rereading passes, allowing MemReread‑4B to outperform strong baselines on long‑context tasks while keeping linear time complexity.
By Baibei Ji, Xiaoyang Weng, Juntao Li, Zecheng Tang, Yihang Lou, Min Zhang
AgroGround is a large‑scale dataset that links plant disease and agricultural target labels to image regions, enabling models to both recognize and localize these targets. It was created by automatically converting eight agricultural VQA datasets into 794,850 grounded instruction examples, including healthy images for negative supervision. Fine‑tuning a shared vision‑language model on these instructions improves recognition, joint accuracy, and healthy‑image abstention, outperforming its annotation teacher on both the AgroGround benchmark and the external PlantSeg test set.
By Abdulla Alshehhi, Zongyan Han, Rao Muhammad Anwer
The paper examines whether a frozen recurrent network can supply sufficient memory for partially observable reinforcement learning tasks, instead of training the recurrent module jointly with the policy. Using a fixed recurrent network paired with a trainable output layer, the authors test this setup on various POMDPs with different memory demands, observation types, and action spaces. Results show that frozen networks can recall information over hundreds of steps and achieve competitive performance while using fewer than half the trainable parameters of fully trainable recurrent models.
SWEAR is a difficulty‑aware reasoning framework for large audio‑language models that derives five‑dimensional difficulty scores and compressed reasoning traces from offline assessments. It learns these difficulty signals during supervised fine‑tuning and employs reinforcement learning to adapt reasoning allocation across stages, shortening reasoning for easy instances while maintaining performance on hard ones. Experiments on MMAU‑test‑mini and MMAR demonstrate that SWEAR improves accuracy and reduces generated tokens.
MotionCoach is a multimodal agent designed to enhance video editing models for professional motion editing, such as sports and martial arts. It generates grounded prompts and reference clips through multi‑turn tool use and self‑verification, enabling precise motion guidance without retraining the base editing models. The system is trained on a curated dataset and evaluated on VPMEBench, achieving significant score improvements on benchmark datasets.
PIVOT introduces a dynamic transition framework for vertical-domain few-shot distillation, routing samples between On-Policy Distillation (OPD) and GRPO-based reinforcement learning based on teacher-evaluated perplexity. Low-perplexity samples are moved to GRPO for reward-driven refinement, while high-perplexity samples remain under OPD to continue acquiring domain knowledge. Experiments on Banking77 and HWU64 demonstrate that PIVOT outperforms both continued OPD and globally synchronized OPD→GRPO baselines, achieving stronger downstream performance and more stable training dynamics.
By Heng Li, Yong Zhang, Ning Cheng, Zhigen Li, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao
AgentFly is an agentic reinforcement learning framework that introduces a unified resource layer to manage heterogeneous environments—such as sandboxes, model services, and external APIs—through a single scheduling engine. The framework is organized into four layers: an agent layer that abstracts agents, tools, and rewards; a rollout layer that composes agent loops and computes rewards; a context layer that organizes rollouts and injects contextual information; and a low‑level resource layer that handles resource management. AgentFly offers prebuilt tools and environments, demonstrates successful agent training across multiple tasks and models, and provides the first controlled cross‑framework throughput comparison against other agentic RL frameworks.
By Renxi Wang, Rifo Ahmad Genadi, Bilal El Bouardi, Yongxin Wang, Fajri Koto, Zhengzhong Liu, Timothy Baldwin, Haonan Li
EvoRubric introduces a co‑evolutionary reinforcement learning framework that learns rubric‑driven evaluation criteria for open‑ended text generation. The system uses a shared policy that simultaneously generates responses and rubrics, while a frozen meta‑verifier and grader provide validity checks and scoring. Experiments on medical, writing, and science benchmarks show that EvoRubric outperforms static and other evolving‑rubric baselines, achieving higher average scores on five tasks.
By Xin Guan, Xiaomeng Hu, Shen Huang, Zhenyi Wang, Bo Zhang, Zijian Li, Pengjun Xie, Bo Liu, Jiuxin Cao
ViSkill is a visual-native skill learning framework that encodes successful interactions into composite visual skill cards for VLM agents. These skill cards guide inference and reward shaping, while distilled trajectories are reintegrated into the library, creating a closed feedback loop that reinforces both skill accumulation and policy improvement. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves a 0.89 success rate (0.91 with cold‑start), outperforming existing baselines and converging faster than standard PPO.
By Hongxing Li, Dingming Li, Yixin Li, Yong Du, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
SkillGraph introduces a skill library framework that models reusable skills as nodes in a directed graph, with typed edges representing prerequisite, enhancement, and co-occurrence relationships. When presented with a new task, the system retrieves an ordered subgraph of relevant skills, guiding multi-step decision making. The graph is continuously refined through agent trajectories and reinforcement learning, enabling simultaneous improvement of the skill library and the agent policy, and achieving state‑of‑the‑art results on ALFWorld, WebShop, and several search‑augmented QA tasks.
By Xiaoyuan Li, Moxin Li, Keqin Bao, Yubo Ma, Wenjie Wang, Dayiheng Liu, Fuli Feng
SERL‑SQL introduces a selective execution‑grounded reinforcement learning framework for multi‑turn Text‑to‑SQL agents. It samples on‑policy interaction trajectories and employs a training‑only teacher to re‑score student actions using execution feedback, converting the teacher‑student likelihood gap into bounded, masked weights that reweight GRPO advantages only on SQL and tool‑action tokens. Experiments on BIRD, Spider, and cross‑domain benchmarks show competitive performance, achieving 76.56% execution accuracy on BIRD‑Dev and 89.92% on Spider‑Test, and the reward‑based selection strategy approaches the oracle Best‑of‑N upper bound.
By Tao Liu, Tao Feng, Xiangheng Li, Jinwang Song, Yifan Li, Xiaoqing Cheng, Dixuan Zhang, Siquan Li, Lin Lan, Hongying Zan, Kunli Zhang, Chao Wu
SPLIT-RL introduces a staged post‑training method that separates visual reasoning (VR) and language reasoning (LR) into distinct phases, allowing each phase to be optimized with a phase‑specific reward. The approach includes Claim‑Level Advantage (CLA‑GRPO), which breaks VR rollouts into atomic visual claims and assigns fine‑grained advantages based on visual‑type group formation. Experiments on Qwen3‑VL and InternVL models show that SPLIT‑RL improves average accuracy by 1.4–6.1 points over GRPO and enhances both VR and LR capabilities.
By Raja Kumar, Rajat Koner, Ritwick Chaudhry, Zhuowei Li, Nishant Sankaran, Yifan Xing