arXiv Computation and Language

Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification

arXiv AI

Jun 30

Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning

arXiv:2606. 29984v1 Announce Type: new Abstract: Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs).

By Peng, Lee, Yin Zhang, Yanglin Zhang, Haonan Wu, Zishan Liu, Ruoxi Zang, Xin Zhu, Jiayin Zheng, Jian Yao, Zefeng Ji, Fei Ma

llms reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 15

Self-Evolving Visual Questioner

arXiv:2606. 13929v1 Announce Type: cross Abstract: Vision-language models (VLMs) are typically trained as passive answerers, while their ability to actively ask diverse, non-trivial, visual-centric and grounded questions remains underexplored.

By Yijun Liang, Hengguang Zhou, Ming Li, Lichen Li, Cho-Jui Hsieh, Tianyi Zhou

llms agents multimodal

arXiv AI

Oct 12

CLEAT: Learning Value-Guided Clarification for Interactive Agents

CLEAT (CLarification through valuE-guided Agent co-Training) is a framework that jointly trains a clarification-value controller and a task policy to improve interactive intent alignment in large language model agents. It introduces a residual query value metric, estimated via paired rollouts, to assess the utility of asking clarification questions and uses this to guide both controller predictions and policy decisions. Experiments with Qwen3-4B show CLEAT outperforming baselines across multiple benchmarks, with significant gains in user‑centric tasks and demonstrating that co‑training benefits the task policy even when the controller is omitted at inference.

By Jiatan Huang, Zheyuan Zhang, Ziming Li, Tianyi Ma, Mingchen Li, Yanfang Ye, Chuxu Zhang

llms agents benchmarks safety

arXiv Computer Vision

Sep 1

Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering

arXiv:2604.07146v3 Announce Type: replace Abstract: Knowledge-based visual question answering (KB-VQA) requires vision-language models to understand images and use external knowledge, especially for...

By Zhuohong Chen, Zhenxian Wu, Yunyao Yu, Hangrui Xu, Zirui Liao, Zhifang Liu, Xiangwen Deng, Pen Jiao, Haoqian Wang

llms rag agents nlp fine-tuning multimodal benchmarks

arXiv Computation and Language

Sep 30

Question-Specific Knowledge Graphs for Efficient Visual Reasoning

arXiv:2609.35942v1 Announce Type: new Abstract: Recent work in visual question answering has shown that vision-language models can exhibit strong reasoning capabilities by translating visual inputs i...

By Ting-Chih Chen, Emile van Krieken, Shujian Yu, Filip Ilievski

llms nlp reinforcement-learning multimodal benchmarks

arXiv AI

Sep 23

When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain

arXiv:2609.24290v1 Announce Type: new Abstract: Instruction-tuned LLMs faced with underspecified queries often commit to a single interpretation rather than ask for clarification, producing confident...

By Yunxiang Li, Xixin Wu, Helen Meng

llms agents reinforcement-learning benchmarks

arXiv AI

Sep 10

Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning

arXiv:2609.08025v1 Announce Type: new Abstract: Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms su...

By Vishwas Sathish, Viresh Ranjan, Xinliang Zhu, Arnab Dhua, Douglas Gray

llms agents nlp reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Oct 15

Self-Evolving Visual Agent via Margin Reward

arXiv:2610.15084v1 Announce Type: new Abstract: Visual agents that interleave reasoning with tool invocation have posed a promising direction for multimodal tasks. However, learning effective tool in...

By Wenxi Yang, Yuzhong Zhao, Baochang Zhang, Yunfan Liu, Qixiang Ye

agents reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jul 7

Incentivizing Vision Language Models to Search for Long Video Question Answering

arXiv:2607. 02959v1 Announce Type: cross Abstract: We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process.

By Harsh Goel, S P Sharan, Sahil Shah, Minkyu Choi, Joungbin An, Kristen Grauman, Sandeep P. Chinchali

llms agents nlp reinforcement-learning benchmarks

arXiv AI

Jun 4

Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning

arXiv:2605. 14054v2 Announce Type: replace Abstract: Achieving robust perception-reasoning synergy is a central goal for advanced Vision-Language Models (VLMs).

By Haozhe Wang, Qixin Xu, Changpeng Wang, Taofeng Xue, Chong Peng, Wenhu Chen, Fangzhen Lin

llms agents reinforcement-learning multimodal

arXiv AI

Jun 3

Uncertainty-Aware Clarification in LLM Agents with Information Gain

arXiv:2606. 03135v1 Announce Type: new Abstract: Large Language Model (LLM) agents often operate under underspecified user instructions, where latent uncertainty over user intent leads to erroneous tool actions.

By Mengyi Deng, Zhiwei Li, Xin Li, Tingyu Zhu, Ying Zhao, Zhijiang Guo, Wei Wang

llms agents

arXiv AI

Sep 30

Rational Clarification by Assistive Agents via Value-of-Information Reasoning

arXiv:2609.37588v1 Announce Type: new Abstract: Users of language-based assistive agents often make ambiguous requests. In response, an assistant can either directly act on its interpretation of the...

By T. Duy Nguyen-Hien, Yee Whye Teh, Wee Sun Lee, Tan Zhi-Xuan

llms agents nlp fine-tuning

Related stories