arXiv Computer Vision

Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?

The paper introduces the Medical Data Standardization Benchmark (MDS‑Bench), which evaluates vision‑language models (VLMs) on their ability to process raw, heterogeneous medical data. Models must identify source formats, convert raw images into VLM‑compatible inputs, extract relevant text, and organize the results into structured image‑text pairs. Experiments show that even the top VLM, Gemini 3 Flash, achieves only a 48.6% end‑to‑end success rate, underscoring the challenge of raw data standardization in clinical settings.

Hugging Face Trending Papers

Jul 6

Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?

As vision-language models (VLMs) are increasingly applied to medical AI, existing benchmarks mainly focus on evaluating their diagnosis ability over given medical images and texts, implicitly assuming that standardized medical images, texts or question-answer pairs are already prepared. However, this assumption does not hold when we apply VLMs in real clinical practice, where medical data is often raw, heterogeneous, and fragmented across different sources.

llms multimodal benchmarks

arXiv AI

Jun 9

MedVision: Benchmarking Quantitative Medical Image Analysis

arXiv:2511. 18676v2 Announce Type: replace-cross Abstract: Current vision-language models (VLMs) in medicine are primarily designed for categorical question answering (e.

By Yongcheng Yao, Yongshuo Zong, Raman Dutt, Yongxin Yang, Sotirios A Tsaftaris, Timothy Hospedales

llms nlp fine-tuning multimodal benchmarks

arXiv AI

Aug 14

Can Generalist Vision Language Models (VLMs) Rival Specialist Medical VLMs? Benchmarking and Strategic Insights

arXiv:2506. 17337v5 Announce Type: replace-cross Abstract: Vision Language Models (VLMs) have shown promise in automating image diagnosis and interpretation in clinical settings.

By Yuan Zhong, Ruinan Jin, Qi Dou, Xiaoxiao Li

llms fine-tuning benchmarks

arXiv AI

Jul 2

Text Over Image: Auditing Multimodal Robustness in Synthetic Medical Image Detection

arXiv:2606. 25375v2 Announce Type: replace-cross Abstract: With the rapid adoption of generative AI, synthetic medical images pose growing risks, including diagnostic deception and insurance fraud.

By Ching-Hao Chiu, Hao-Wei Chung, Gelei Xu, Xueyang Li, Pin-Yu Chen, John Kheir, Meysam Ghaffari, Carlos Morato, Ahmed Abbasi, Yiyu Shi

llms multimodal benchmarks

arXiv Computer Vision

Aug 25

MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

arXiv:2608.22323v1 Announce Type: new Abstract: The application of Large Language Models (LLMs) to diagnostic decision-making has garnered growing interest. However, existing benchmarks largely focus...

By Lai Wei, Yuchao Chen, Zhenbiao Cao, Xiaojin Zhang, Zhongyu Wei, Bangting Wang, Wei Chen, Xiang Bai

llms multimodal benchmarks

arXiv Computer Vision

Aug 25

Expert-level vision-language foundation model for real-world radiology and comprehensive evaluation

arXiv:2409.16183v2 Announce Type: replace Abstract: Radiology is a vital and complex component of modern clinical workflow and covers many tasks. Recently, vision-language (VL) foundation models in m...

By Xiaohong Liu, Guoxing Yang, Yulin Luo, Jiaji Mao, Xiang Zhang, Haibo Wang, Zhiyang He, Ming Gao, Shanghang Zhang, Jun Shen, Guangyu Wang

multimodal benchmarks

arXiv Machine Learning

Jul 10

A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding

arXiv:2512. 21414v2 Announce Type: replace-cross Abstract: Recent tool-use frameworks powered by vision-language models (VLMs) improve image understanding by grounding model predictions with specialized tools.

By Christina Liu, Alan Q. Wang, Joy Hsu, Jiajun Wu, Ehsan Adeli

llms multimodal benchmarks

arXiv AI

Jul 3

Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias

arXiv:2607. 01973v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are increasingly applied in medical tasks such as pathology description, report generation, and visual question answering.

By Sofiane Ouaari, Kevin Vorwalder, Nico Pfeifer

llms rag nlp multimodal benchmarks safety

arXiv AI

Jul 7

Evaluating and Understanding Model Editing for Medical Vision Language Models

arXiv:2607. 05310v1 Announce Type: new Abstract: Model editing promises a fast, targeted way to correct post-deployment mistakes in medical vision-language models (VLMs) without costly retraining.

By Guli Zhu, Chenwei Wu, Liyue Shen

llms diffusion multimodal benchmarks

arXiv AI

Sep 10

MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Data

arXiv:2601.06847v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) can generate convincing clinical narratives, yet frequently struggle to visually ground their statements. We po...

By Mengmeng Zhang, Xiaoping Wu, Hao Luo, Fan Wang, Yisheng Lv

llms computer-vision multimodal

arXiv Machine Learning

Jul 9

MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models

arXiv:2607. 07673v1 Announce Type: cross Abstract: Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams.

By Hyunjae Kim, Dain Kim, Pan Xiao, Serina S. Applebaum, Younjoon Chung, Xuguang Ai, Yu Yin, Roy Jiang, Yuexi Du, Yawen Wei, Yiming Kong, Tuo Guo, Zhiyuan Cao, Mengmeng Du, Yuelei Fu, Yan Hu, Rui Shi, Gui Yang, Kevin W. Jin, Yuntian Liu, Yuxuan Tian, Jonathan Marquez, Zhen Chen, Sheng Zhang, Hoifung Poon, Hua Xu, Jaewoo Kang, Qingyu Chen

llms multimodal benchmarks safety

arXiv AI

Jul 3

MedRepBench: A Comprehensive Benchmark for Medical Report Interpretation

arXiv:2508. 16674v2 Announce Type: replace-cross Abstract: Medical report understanding from real-world document images is essential for generating patient-facing explanations and enabling structured information exchange in clinical systems.

By Fangxin Shang, Yuan Xia, Dalu Yang, Yahui Wang, Binglin Yang

llms computer-vision multimodal benchmarks safety

Related stories