arXiv Computer Vision

MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

MMSearch-OPD is a new approach for long‑horizon multimodal search agents that combines a controllable data refinement pipeline with joint reinforcement learning and on‑policy distillation. The refinement process expands reasoning chains from existing simple multimodal questions, allowing adjustable search depth and cross‑modal evidence requirements while reusing image–question pairs. Training uses answer rewards, behavior penalties, and a decaying distillation coefficient to guide the agent, and experiments demonstrate that both data refinement and joint training after supervised fine‑tuning improve long‑horizon multimodal search performance.

arXiv AI

Aug 26

OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning

OPDSearch+ introduces a two‑stage distillation framework for search‑augmented reasoning that eliminates the need for task‑specific teacher fine‑tuning. In the first stage, a frozen off‑the‑shelf instruct model guides a student through live search interactions using a per‑position forward KL objective, transferring reasoning decomposition and evidence integration skills. The second stage refines this student with reinforcement learning, achieving performance surpassing RL alone and outperforming all prior 3B‑parameter baselines on seven QA benchmarks, including 13.1% improvement on HotpotQA and 8.5% on 2WikiMultihopQA.

By Qinglin Ye, Zhiyuan Gu, Jingjie Xia, Yiheng Zhang, Kaiyan Zhao, Shunchao Zheng, Yuhang Mu, Wenchao Du, Yiming Wang

llms reinforcement-learning fine-tuning efficiency benchmarks

Hugging Face Trending Papers

Aug 25

OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning

Search-augmented reasoning remains difficult for small language models. On-policy distillation (OPD) from trained teachers offers a promising direction, but suffers from two issues: (1) high-quality m...

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jun 8

Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization

arXiv:2606. 07000v1 Announce Type: new Abstract: Recent post-training methods, particularly Reinforcement Learning with Verifiable Rewards (RLVR), have significantly enhanced the reasoning ability of Large Vision-Language Models (LVLMs).

By Shizhe Xiang, Ke An, Wenlong Yu, Yue Liu, Jian Luan, Pei Fu, Qilong Wang

llms reinforcement-learning efficiency multimodal safety

arXiv AI

Oct 2

Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents

Selection-Based Structured Reasoning (SSR) is a framework that replaces free-form reasoning with the selection of pre-specified natural‑language reasoning candidates, allowing multimodal agents to choose from a set of reusable high‑level reasoning traces. By scoring these candidates in parallel using a shared context KV cache, SSR eliminates the need for an auxiliary task head and reduces inference cost. Experiments on seven multimodal search benchmarks with 2B and 4B models show that SSR maintains competitive success rates while cutting per‑turn reasoning latency by over 90% and overall inference latency by 28–54%.

By Feiyu Gavin Zhu, Xiaoyu Zhu, Jiqi Yang, Rui Yang, Arnab Kumar Mondal, Yancheng Wang, Xinke Deng, Jean Oh, Reid Simmons, Joerg Liebelt, Xiang Kong, Zhongyu Jiang

agents reinforcement-learning fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Jun 5

ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation

arXiv:2606. 05718v1 Announce Type: cross Abstract: On-policy distillation (OPD) improves reasoning by training a student on trajectories sampled from its own policy under supervision from a teacher.

By Kanghui Tian, Siyuan Liu, Ziang Yan, Sheng Xia, Shuai Dong, Yi Wang

efficiency multimodal benchmarks

arXiv Computer Vision

Oct 1

Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillation

arXiv:2609.39120v1 Announce Type: new Abstract: On-policy distillation (OPD) improves reasoning by providing token-level supervision from a teacher on a student's own trajectories. Existing methods p...

By Siyuan Liu, Kanghui Tian, Yue Duan, Yutao He, Shangdong Yang, Jian Zhang, Yinghuan Shi

efficiency multimodal benchmarks

arXiv AI

Sep 10

Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning

arXiv:2609.08025v1 Announce Type: new Abstract: Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms su...

By Vishwas Sathish, Viresh Ranjan, Xinliang Zhu, Arnab Dhua, Douglas Gray

llms agents nlp reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Sep 16

Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation

arXiv:2609.16059v1 Announce Type: cross Abstract: Multimodal instruction following (MMIF) is crucial for building generalist agents. However, current training paradigms rely heavily on Supervised Fin...

By Yirong Zeng, Zhang Sai, Yuxian Wang, Yutai Hou, Yufei Liu, Xiao Ding, Bibo Cai

agents reinforcement-learning fine-tuning efficiency multimodal benchmarks

arXiv AI

Jul 28

From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search

arXiv:2607. 24280v1 Announce Type: new Abstract: Agentic search enables large language models to solve knowledge-intensive tasks by interleaving multi-step reasoning with retrieval, yet optimizing this with outcome-based reinforcement learning (RL) provides only sparse supervision.

By Junlin Liu, Jiangwang Chen, Zixin Song, Shuaiyu Zhou, Chunji Lv, Hank Wu, Kailin Jiang, Jinyang Wu, Bohan Yu, Chenxi Zhou

llms agents reinforcement-learning efficiency benchmarks

arXiv AI

Jun 29

ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering

arXiv:2606. 27974v1 Announce Type: cross Abstract: Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge.

By ZhengXian Wu, Hangrui Xu, Kai Shi, Zhuohong Chen, Yunyao Yu, Chuanrui Zhang, Zirui Liao, Jun Yang, Zhenyu Yang, Haonan Lu, Haoqian Wang

rag agents nlp reinforcement-learning fine-tuning multimodal

arXiv AI

5d ago

Neos: Learning to Interleave Search and Visual Reasoning

arXiv:2610.17043v1 Announce Type: new Abstract: Recent multimodal deep search agents have advanced tool use and visual content retrieval, yet they still struggle to convert intermediate visual eviden...

By Ekko He, Jack Hong, Shilin Yan, Xintong Zhang, Yujie Zhang, Wenjie Li, Haoran Sun, Kaixun Jiang, Jiyuan Fu, Guannan Zhang, Hongwei Xue

agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Aug 18

Deep Thought Alignment: Trajectory-Level Latent Distillation for Video Reasoning

arXiv:2608. 16316v1 Announce Type: cross Abstract: Large Multimodal Models (LMMs) for video reasoning have long been hindered by the high computational cost of processing vast amounts of visual information.

By Ao Shen, Yongheng Zhang, Yinghui Li, Manning Wang, Di Yin, Xing Sun

efficiency multimodal benchmarks safety

Related stories