Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,018 stories · RSS feed

arXiv AI

Aug 5

GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

arXiv:2608. 03270v1 Announce Type: cross Abstract: GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents.

By Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li, Yimin Deng, Jingtong Gao, Junjia Qi, Hanyu Yan, Yefeng Zheng, Xiaopeng Li, Wanyu Wang, Xian Wu, Xiangyu Zhao

llms agents computer-vision multimodal benchmarks

arXiv AI

Aug 5

OliveGemma: A 3 Billion Visual Language Model for Recognising the Mediterranean & European Diet

arXiv:2608. 03428v1 Announce Type: cross Abstract: Image based dietary assessment offers a scalable alternative to self reported food diaries, yet fine-grained food recognition remains challenging due to high intra-class variability and visually similar dishes.

By Dimitrios I. Zaridis, Traianos Tsiokris, Vasileios C. Pezoulas, Daphni Plati, Eugenia Mylona, Eleni Georga, Nikos Tsiknakis, Antonis Sakellarios, Dimitrios I. Fotiadis

llms fine-tuning multimodal

arXiv AI

Aug 5

Continue or Replan? Bernoulli-Continuation Policy Learning for Adaptive Horizon Execution

arXiv:2608. 03483v1 Announce Type: cross Abstract: Existing chunk-based Vision-Language-Action (VLA) models execute a fixed number of actions (i.

By Weichen Xu, Zhenhua Liu, Lin Luo, Yaobo Liang, Chengtang Yao, Qingyu Mei, Jian Cao, Xixin Cao, Xing Zhang, Jiaolong Yang, Baining Guo

reinforcement-learning robotics multimodal safety

arXiv AI

Aug 5

AI-Based Sound Effect Generation: A Narrative Review of Generative Models Across Input Modalities

arXiv:2608. 03742v1 Announce Type: cross Abstract: Sound effects play a crucial role in conveying actions, events, and environmental cues across digital applications, often requiring a high degree of variation and contextual adaptability.

By Sandy Abdo, Bill Kapralos, Priyamvada Tripathi, KC Collins, Adam Dubrowski

multimodal benchmarks

arXiv AI

Aug 5

Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

arXiv:2608. 03979v1 Announce Type: cross Abstract: We introduce Video-DeepResearch (Video-DR), extending multimodal agents from static images to continuous video streams, a setting that demands dense spatiotemporal grounding coupled with open-web exploration.

By Zhen Fang, Yu Zeng, Wenxuan Huang, Yiming Zhao, Shiting Huang, Tianfei Ren, Qi Lu, Qingnan Ren, Qisheng Su, Lionel Z. Wang, Qingyu Yin, Shuang Chen, Zehui Chen, Lin Chen, Zhenfei Yin, Yao Hu, Shaohui Lin, Wanli Ouyang, Shaosheng Cao, Feng Zhao

llms agents fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Aug 5

Attention is Case-Sensitive

arXiv:2608. 03711v1 Announce Type: cross Abstract: In human visual perception, uppercase lettering serves as a natural salience cue that captures attention within lowercase text.

By Maximilian Dillitzer, Tin Stribor Sohn, Jason J. Corso, Michael Auerbach

llms nlp fine-tuning multimodal

arXiv AI

Aug 5

CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

arXiv:2608. 03890v1 Announce Type: cross Abstract: A clinically useful chest X-ray system must go beyond fluent report generation: it should classify findings with tunable decision thresholds, localize them spatially, and derive the anatomical measurements upon which many diagnoses depend.

By Mercy Prasanna Ranjit, Anirban Porya, Sathvik Joel, Niharika Vadlamudi, Nikhilesh Chowdary Eathamukkala, Prasanth V V, Abhyuday Kumara Swamy, Pranay Narhari Umredkar, Pradeep Narayan, Vivek Rajagopal, Tanuja Ganu

llms nlp multimodal benchmarks

arXiv AI

Aug 5

DiffImaginE: Imagine to Verify Entity Types with Diffusio

arXiv:2608. 03025v1 Announce Type: new Abstract: Multimodal named entity recognition (MNER) determines whether each candidate span and entity-type hypothesis is supported by joint textual and visual evidence.

By Feng Zhang, Feiyu Han, Rongxin Yang, Yang Liu, Yancheng Chen, Rui Wang, Yingguang Yang, Tian Xueyun, Chongyang Zhang, Hao Zheng, Xu Kefu, Congjing Ran, Fuhai Chen, Bin Chong

diffusion nlp multimodal

arXiv AI

Aug 5

Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation

arXiv:2608. 03791v1 Announce Type: new Abstract: Vision-Language Models (VLMs), like Large Language Models (LLMs), may memorize sensitive, copyrighted, or harmful knowledge from their pretraining corpora.

By Chunlin Liu, Junnian Chen, Haitong Jiang, Jianyu Zhao, Yingsen Pang, Jingchen Li, Jiabiao He, Youming Lu, Jinhe Bi, Yuntao Du

llms multimodal benchmarks

arXiv AI

Aug 5

Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs

arXiv:2608. 03450v1 Announce Type: cross Abstract: Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction.

By Haoqian Kang, Liupeng Li, Kuofeng Gao, Jinpeng Wang, Zhenyu Lu, Bin Chen, Ke Chen, Yaowei Wang

llms multimodal benchmarks safety

arXiv AI

Aug 5

UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space

arXiv:2608. 03817v1 Announce Type: cross Abstract: Large vision--language models (LVLMs) demonstrate strong multimodal reasoning capabilities but remain prone to hallucination, where model predictions are not grounded in visual evidence.

By Amir Mohammad Ezzati, Kiyan Rezaee, Bardiya Kariminia, Mohamad Amin Yousefi, Asal Mohammadjafari Mamaqani, Behrad Samimi, Mohammad Hossein Rohban

llms multimodal benchmarks safety

arXiv AI

Aug 5

When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding

arXiv:2608. 03918v1 Announce Type: cross Abstract: Efficient long-video understanding requires vision--language models (VLMs) to reason over a small number of frames selected as sparse visual evidence.

By Ke Li, Jiayu Chen, Maoliang Li, Zihao Zheng, Hailong Zou, Hengyi Zhang, Xuanzhe Liu, Xiang Chen

llms agents multimodal

arXiv AI

Aug 5

From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Model

arXiv:2508. 00955v3 Announce Type: replace-cross Abstract: Adapting generative Multimodal Large Language Models (MLLMs) into universal embedding models typically demands resource-intensive contrastive pre-training, while traditional hard negative mining methods suffer from severe false negative contamination.

By Yeong-Joon Ju, Seong-Whan Lee

llms rag fine-tuning multimodal benchmarks

arXiv Machine Learning

Aug 5

PatTree: a novel approach for automated creation of multimodal, graph-based patient representations for medical classification tasks

arXiv:2608. 02692v1 Announce Type: new Abstract: Access to holistic, multimodal data improves the performance of Artificial Intelligence (AI) in medical classification tasks compared to utilizing single modalities or data sources.

By Julia Gehrmann, Lars Quakulinski, Hamza Naseem, Oya Beyan

multimodal benchmarks

arXiv Machine Learning

Aug 5

Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering

arXiv:2608. 02907v1 Announce Type: new Abstract: Multimodal retrievers are essential for knowledge-based visual question answering, where they retrieve external evidence for image-question pairs.

By Jingchen Sun, Shaobo Han, Ruiyi Zhang, Naresh Kumar Devulapally, Ming Liu, Yitao Long, Vishnu Suresh Lokhande, Changyou Chen

nlp multimodal benchmarks

arXiv AI

Aug 5

Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement

arXiv:2608. 03733v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved remarkable performance across vision-language tasks, but their progress depends heavily on large-scale, high-quality multimodal data that are costly to annotate.

By Chunyang Jiang, Pingping Zhang, Yuzhi Zhao, Wenao Ma, Zhijian Hou, Mengyang Wu, Yiyang Cai, Senkang Hu, Sitong Cheng, Chi-Min Chan, Wei Xue, Yike Guo

llms nlp multimodal benchmarks

arXiv AI

Aug 5

Taming the Implicit: Dual-Channel Risk-Aware Reinforcement Fine-Tuning for Continual Multimodal Post-Training

arXiv:2608. 03660v1 Announce Type: new Abstract: Reinforcement fine-tuning (RFT) is widely believed to inherently resist catastrophic forgetting in continual post-training of multimodal large language models.

By Yibei Liu, Jiajun Chen, Qianle Zhang, Tangyue Jin, Mengying Zhu, Meng Xi, Yangyang Wu

llms fine-tuning multimodal benchmarks

arXiv AI

Aug 5

SAGE: Semantic Explainability of Attention-Based Survival Models in Computational Pathology

arXiv:2608. 02803v1 Announce Type: cross Abstract: Attention-based multiple instance learning (ABMIL) is the predominant approach for slide-level prediction in computational pathology, yet its attention maps provide only local explanations: they indicate where a model focuses but not which histological features drive its predictions or how the model behaves across a patient cohort.

By Abdallah Lamane, Abdul Rahman Diab, Ren-Chin Wu, William Lotter

llms multimodal benchmarks

arXiv AI

Aug 5

Chat Debugging: An Exploratory Study of Human-AI Collaboration to Debug Analog Circuits

arXiv:2608. 02955v1 Announce Type: cross Abstract: This research paper describes an exploratory study on the effectiveness of Chat Debugging: troubleshooting malfunctioning analog circuits on breadboards and printed circuit boards (PCB) by undergraduates through conversations with public-domain large language models (LLMs).

By John Hu, Andrew Ash

llms multimodal

arXiv AI

Aug 5

Standalone DINOv3 for Training-Free Open-Vocabulary Semantic Segmentation in Remote Sensing

arXiv:2608. 03023v1 Announce Type: cross Abstract: Remote sensing semantic segmentation is hindered by costly pixel-level annotations, motivating training-free open-vocabulary methods.

By Changhao Zhao, Haoxiang Li, Yuke Li, Hai Liu, LingLin Zeng

computer-vision fine-tuning multimodal

Get the daily briefing