Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,954 stories · RSS feed

arXiv Machine Learning

Jul 21

SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis

arXiv:2607. 18046v1 Announce Type: new Abstract: Graphical User Interface (GUI) agents powered by vision-language models hold promise for automating real-world mobile tasks.

By Zhuohang Fan, Beichen Zhang, Yuanfa Li, Changqiao Wu, Wei Liu, Jian Luan, Weigang Zhang

llms agents fine-tuning multimodal

arXiv Machine Learning

Jul 21

Precision-Varying Prediction (PVP): Robustifying ASR systems against adversarial attacks

arXiv:2603. 22590v2 Announce Type: replace Abstract: With the increasing deployment of automated and agentic systems, ensuring the adversarial robustness of automatic speech recognition (ASR) models has become highly relevant.

By Mat\'ias Pizarro, Raghavan Narasimhan, Jonas Killian, Asja Fischer

agents multimodal safety

arXiv AI

Jul 21

An Explicit World Model Based on Data-First Ontology: DaoQL Multimodal Storage Validation and Counterfactual Reasoning Evaluation

arXiv:2607. 17269v1 Announce Type: new Abstract: Large language models encode world models implicitly in neural weights, which exposes four structural risks in high-precision domains such as medicine and finance: hallucination, frozen knowledge, poor explainability, and poor modifiability.

By Zhanbo Li, Shifeng Wu, Xiangjin Meng, Wenjie Cai

llms agents multimodal benchmarks safety

arXiv AI

Jul 21

Constraint-Anchored Reasoning Traces

arXiv:2607. 16727v1 Announce Type: new Abstract: Autoregressive multimodal large language models (MLLMs) suffer from error snowballing: a single incorrect inference early in a chainof-thought (CoT) trace corrupts all downstream reasoning.

By Zehua Cheng, Wei Dai, Jiahao Sun

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 21

FUSAR-R1: A Large-Scale Reasoning Model for Intelligent Interpretation of SAR Images

arXiv:2607. 16819v1 Announce Type: new Abstract: In recent years, large-scale vision-language models have been driving a paradigm shift in intelligent remote sensing image interpretation.

By Yi Yang, Xiaokun Zhang, Yuxuan Li, Ruyi Zhang, Xinpeng Zhou, Haipeng Wang

llms reinforcement-learning multimodal

arXiv AI

Jul 21

GenSyn10: A Multi-Generative AI Dataset For Benchmarking Image Classification

arXiv:2607. 16283v1 Announce Type: cross Abstract: The rapid advancement of generative AI has outpaced our ability to reliably detect its outputs, particularly when detectors encounter generators they have not seen before.

By Md Faraz Kabir Khan, Saeed Anwar, Ghulam Mubashar Hassan

llms diffusion computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jul 21

SynGR: Unleashing the Potential of Cross-Modal Synergy for Generative Recommendation

arXiv:2605. 18920v2 Announce Type: replace-cross Abstract: Generative Recommendation (GR) has emerged as a promising paradigm by formulating item recommendation as a sequence-to-sequence generation task over item identifiers.

By Wei Chen, Xingyu Guo, Shuang Li, Fuwei Zhang, Meng Yuan, Jing Fan, Zhao Zhang, Deqing Wang, Fuzhen Zhuang

multimodal benchmarks safety

arXiv Machine Learning

Jul 21

The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

arXiv:2607. 18237v1 Announce Type: cross Abstract: Human visual similarity judgments are context-dependent.

By Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann, Jun-Yan Zhu, Eli Shechtman, Alexei A. Efros, Richard Zhang

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 21

O-VAD: Industrial Video Anomaly Detection through Object-Centric Tracking and Reasoning

arXiv:2607. 18142v1 Announce Type: cross Abstract: Industrial Video Anomaly Detection (IVAD) aims to identify anomalous objects and events in an industrial process, which is crucial for modern manufacturing and quality control systems.

By Mei Yuan, Qi Long, Qifeng Wu, Zhenyang Li, Yizhou Zhao, Lei Wang, Yang Liu, Min Xu

agents fine-tuning multimodal

arXiv AI

Jul 21

LenGuard-GPC: Length Guarding with Guided-Prompt Consistency for Spatial Reasoning Reinforce Learning

arXiv:2607. 17243v1 Announce Type: new Abstract: Multi-view spatial reasoning requires vision-language models to compare visual evidence across images, align object correspondences, and infer spatial relations over long visual contexts, a setting where chain-of-thought reasoning tends to grow verbose without becoming more accurate.

By Xingjian Tao, Yiwei Wang, Yujun Cai, Jing Tang

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jul 21

Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances

arXiv:2405. 12775v2 Announce Type: replace-cross Abstract: Discovering the semantics of multimodal utterances is essential for understanding human language and enhancing human-machine interactions.

By Hanlei Zhang, Hua Xu, Fei Long, Xin Wang, Kai Gao

multimodal benchmarks

arXiv AI

Jul 21

Geometry-Enhanced Portion Estimation for Multimodal LLMs

arXiv:2607. 16514v1 Announce Type: cross Abstract: Image-based dietary assessment promises to replace costly, bias-prone manual recalls, but portion estimation remains a major blocker.

By Lin Liao, Peng Li

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 21

Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillation

arXiv:2607. 16303v1 Announce Type: cross Abstract: Medical Vision-Language Models (Med-VLMs) require reliable reasoning from fine-grained visual evidence, yet existing models can produce plausible clinical answers by relying on language priors or medical templates rather than truly attending to diagnosis-critical regions.

By Yunhang Qian, Jiaquan Yu, Jiawei Liu, Meng Wang, Hongwei Bran Li, Xiaobin Hu

llms fine-tuning efficiency multimodal safety

arXiv Machine Learning

Jul 21

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

arXiv:2607. 18171v1 Announce Type: new Abstract: Real-time multimodal applications, including voice agents and interactive video generation, compose heterogeneous models into pipelines whose efficient deployment requires application-specific decisions about placement, streaming, and intra-model parallelism.

By Krish Agarwal, Zhuoming Chen, Yanyuan Qin, Zhenyu Gu, Atri Rudra, Beidi Chen

llms agents diffusion efficiency multimodal benchmarks

arXiv AI

Jul 21

Reward-Driven LLM Agent Workflows: Synthesizing POMDP Routing and Self-Correction for Autonomous Decision-Making

arXiv:2607. 17038v1 Announce Type: new Abstract: This paper addresses key technical challenges in current large language model (LLM) agent applications, including long-horizon planning, sparse reward attribution, and dynamic environmental interaction, by designing and optimizing an intelligent agent workflow.

By Amez Amanj Ali, Kuo-Kun Tseng

llms agents reinforcement-learning robotics multimodal benchmarks safety

arXiv AI

Jul 21

Pailitao-MMSearch: Building Native E-Commerce Multimodal Search Foundation

arXiv:2607. 17499v1 Announce Type: new Abstract: The evolution of e-commerce has fundamentally transformed how users search for products, shifting from simple text-based keyword queries to complex multimodal interactions that seamlessly combine product images, natural language descriptions, and mixed-intent instructions.

By Xiaohan Ye, Xu Chen, Zihan Gong, Jian Ding, Lianyu Du, Baicheng Chen, Yunmeng Shu, Jingqian Zhao, Zhixiang Zhao, Shuaiqi Jia, Chong Ma, Shuwen Xiao, Xiangheng Kong, Yuan Gao, Jun Song, Jinsong Lan, Xiaoyong Zhu, Bo Zheng

llms multimodal

arXiv AI

Jul 21

RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants

arXiv:2607. 16243v1 Announce Type: cross Abstract: Multimodal industrial anomaly inspection assistants are a critical component of next-generation smart factories, enabling interactive vision-language-based querying.

By Anushiya Arunan, Xin Li, Yan Qin, U-Xuan Tan, Nhu Khue Vuong, Xiaoli Li, Chau Yuen

llms multimodal benchmarks safety

arXiv AI

Jul 21

Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs

arXiv:2607. 16311v1 Announce Type: cross Abstract: Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.

By Jingyu Sun, Jiachen Tu, Yuyang Xue, Yaoxin Jiang, Guoyi Xu, Zhengtao Yao, Rui Qian, Yizheng Sun, Hongpeng Zhou, Jingyuan Sun, Yan Lin

llms agents multimodal benchmarks

arXiv AI

Jul 21

A Survey on Knowledge-Oriented Retrieval-Augmented Generation

arXiv:2503. 10677v3 Announce Type: replace-cross Abstract: Retrieval-Augmented Generation (RAG) has gained significant attention in recent years for its potential to enhance natural language understanding and generation by combining large-scale retrieval systems with generative models.

By Mingyue Cheng, Yucong Luo, Jie Ouyang, Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, Daoyu Wang, Enhong Chen

rag nlp multimodal benchmarks safety

arXiv AI

Jul 21

Zoom In, Reason Out: Efficient Far-field Anomaly Detection in Expressway Surveillance Videos via Focused VLM Reasoning Guided by Bayesian Inference

arXiv:2604. 23724v3 Announce Type: replace-cross Abstract: Expressway video anomaly detection is essential for safety management.

By Xiaowei Mao, Bowen Sui, Weijie Zhang, Yawen Yang, Shengnan Guo, Shilong Zhao, Jiaqi Lin, Tingrui Wu, Youfang Lin, Huaiyu Wan

llms multimodal safety

Get the daily briefing