← Back to all news
arXiv Computer Vision By Tianjie Ju, Xinyue Xu, Wanxuan Sun, Lingxiao Diao, Gongshen Liu, Zhuosheng Zhang, Cheng Yang

First Things First: Teaching LLM-Based Agents to Prioritize Must-Haves before Nice-to-Haves

Read the original on arXiv Computer Vision →

The paper introduces the problem of teaching multimodal large language models (MLLMs) to prioritize must‑have requirements over nice‑to‑have ones in autonomous agent tasks. It evaluates current MLLMs on 3,649 realistic service‑scenario problems and finds widespread failures, then proposes First Things First Reinforcement Learning (FTF‑rl) to explicitly optimize reasoning over multi‑priority user requirements, achieving significant performance gains and generalizing to logical and mathematical reasoning benchmarks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
4d ago

Beacon: Knowing When and How to Perform Agentic Visual Reasoning

Beacon is a new agentic visual reasoning model that improves multimodal large language models (MLLMs) by better deciding when to use tools and how to use them. It introduces two key concepts—Mode Adaptiveness, which ensures tools are invoked only when necessary, and Tool Effect, which measures the net benefit of tool use— and trains the model with supervised fine‑tuning and reinforcement learning that rewards necessity-aware decisions and expands capability through expert hints. Across 13 benchmarks, Beacon outperforms other open‑source models, achieving the highest average score and the largest net tool‑gain on diagnostic tests.

By Qixun Wang, Yang Shi, Letian Cheng, Zhuoran Zhang, Yan He, Yuqi Tang, Qi Zhang, Xinlei Yu, Ruizhe Chen, Tianrun Xu, Yuanxing Zhang, Pengfei Wan, Haotian Wang, Xianghua Ying
llmsagentsreinforcement-learningfine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Aug 3

Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents

arXiv:2512. 03438v3 Announce Type: replace Abstract: Agentic reasoning models trained with multimodal reinforcement learning (MMRL) have become increasingly capable, yet they are almost universally optimized using sparse, outcome-based rewards computed based on the final answers.

By Reuben Tan, Baolin Peng, Zhengyuan Yang, Hao Cheng, Oier Mees, Theodore Zhao, Andrea Tupini, Isar Meijer, Qianhui Wu, Yuncong Yang, Lars Liden, Yu Gu, Sheng Zhang, Xiaodong Liu, Lijuan Wang, Marc Pollefeys, Yong Jae Lee, Jianfeng Gao
agentsreinforcement-learningroboticsfine-tuningmultimodalbenchmarkssafety
More like this →
arXiv AI
Aug 28

Don't Overthink, Don't Underthink: Toward Adaptive Reasoning in Agentic AI

The paper argues that large language models need adaptive reasoning rather than fixed reasoning budgets. It shows that over‑reasoning leads to high computational cost without accuracy gains, while under‑reasoning results in incorrect or incomplete solutions. The authors evaluate these failure modes on MATH‑500 and the GAIA benchmark, highlighting the need for dynamic reasoning allocation in agentic AI systems.

By Md Jueal Mia, M. Hadi Amini
llmsagentsbenchmarks
More like this →