arXiv Machine Learning

Medical Image Spatial Grounding with Semantic Sampling

arXiv:2603. 14579v3 Announce Type: replace-cross Abstract: Vision language models (VLMs) have shown significant promise in visual grounding for images as well as videos.

arXiv Machine Learning

Jun 19

Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology

arXiv:2606. 20477v1 Announce Type: cross Abstract: We study how to train visually grounded vision-language models (VLMs) for radiology without manual spatial annotations.

By Yusuf Salcan (Computer Vision Group, University of Freiburg, Germany, CRIION-AI Lab, Freiburg, Germany), Simon Ging (Computer Vision Group, University of Freiburg, Germany, Adaptive & Agentic AI), Robin Schirrmeister (Department of Radiology, Medical Center -- University of Freiburg, Germany), Philipp Arnold (Department of Radiology, Medical Center -- University of Freiburg, Germany), Elmar Kotter (Department of Radiology, Medical Center -- University of Freiburg, Germany), Behzad Bozorgtabar (Adaptive & Agentic AI), Thomas Brox (Computer Vision Group, University of Freiburg, Germany)

llms computer-vision nlp fine-tuning multimodal benchmarks

arXiv AI

Aug 11

MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation

arXiv:2608. 09818v1 Announce Type: cross Abstract: Reliable medical image understanding requires models to connect clinical language and visual reasoning with pixel-level grounding.

By Haoyu Yang, Meixing Shi, Zengjie Chen, Haoran Sun, Haitao Leng, Xiaoming Shi, Yuxiang Cai, Yankai Jiang

llms computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jun 16

Thinking with Visual Grounding

arXiv:2606. 16122v1 Announce Type: new Abstract: Visual thinking should not only sound right; it should show its evidence.

By Junkai Zhang, Yihe Deng, Kai-Wei Chang, Wei Wang

llms agents reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Aug 4

Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMs

arXiv:2608. 01473v1 Announce Type: cross Abstract: Multimodal large language models (MLLM) for surgical scene understanding typically inject hundreds of dense visual tokens into a language model, leading to costly inference and limited spatial traceability for generated answers.

By Guiqiu Liao, Matjaz Jogan, Daniel A. Hashimoto

llms nlp multimodal benchmarks safety

arXiv AI

Jul 29

Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models

arXiv:2604. 27720v2 Announce Type: replace Abstract: Vision-language models (VLMs) are increasingly applied to medical visual question answering (Med-VQA), yet whether they can \emph{localize} the evidence behind their answers---a prerequisite for clinical auditability---is poorly characterized.

By Xupeng Chen, Binbin Shi, Chenqian Le, Qifu Yin, Lang Lin, Haowei Ni, Ran Gong, Panfeng Li

llms nlp fine-tuning multimodal benchmarks

arXiv AI

Jul 8

Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning

arXiv:2607. 05798v1 Announce Type: cross Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have evolved from static perception to interleaved visual-language reasoning, often referred to as ``thinking with images''.

By Yake Wei, Yuan Wang, Fengyun Rao, Jing Lyu, Di Hu

llms rag computer-vision multimodal benchmarks safety

Hugging Face Trending Papers

Jun 29

SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation

Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness. However, such metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image.

llms multimodal benchmarks

arXiv AI

Jun 2

CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Models

arXiv:2605. 13178v2 Announce Type: replace-cross Abstract: In large vision-language models, visual tokens typically constitute the majority of input tokens, leading to substantial computational overhead.

By Sangin Lee, Yukyung Choi

llms fine-tuning efficiency multimodal

arXiv Machine Learning

Jun 8

The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models

arXiv:2603. 22278v2 Announce Type: replace-cross Abstract: Many multimodal tasks, such as image captioning and visual question answering, require vision-language models (VLMs) to bind objects with their properties and spatial relations.

By Kelly Cui, Nikhil Prakash, Shoval Messica, Ayush Raina, David Bau, Antonio Torralba, Tamar Rott Shaham

llms nlp multimodal

Hugging Face Trending Papers

Jul 7

Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning

Recent advancements in Multimodal Large Language Models (MLLMs) have evolved from static perception to interleaved visual-language reasoning, often referred to as ``thinking with images''. A basic operation in this reasoning process is to zoom in on regions of interest (often represented with bounding boxes) to acquire finer visual details.

llms rag computer-vision multimodal benchmarks safety

arXiv AI

Aug 5

GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

arXiv:2608. 03270v1 Announce Type: cross Abstract: GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents.

By Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li, Yimin Deng, Jingtong Gao, Junjia Qi, Hanyu Yan, Yefeng Zheng, Xiaopeng Li, Wanyu Wang, Xian Wu, Xiangyu Zhao

llms agents computer-vision multimodal benchmarks

arXiv AI

Jun 29

SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motion

arXiv:2606. 27876v1 Announce Type: cross Abstract: Spatial intelligence is essential for low-altitude unmanned aerial vehicle (UAV) perception, collaboration, and navigation.

By Haoyu Zhang, Meng Liu, Qianlong Xiang, Kun Wang, Yaowei Wang, Liqiang Nie

llms multimodal benchmarks

Related stories