arXiv Computer Vision
Oct 8

Geometry-Supervised Visual Representation Learning for Multi-Phenotype Lesion Interpretation in Medical VLMs

The paper introduces PureVision, a geometry‑supervised framework for medical vision‑language models that tackles the challenge of interpreting multi‑phenotype lesions. It comprises PureEyes, which imposes geometric constraints to preserve anatomical hierarchies and phenotypic subclass relationships, and PureNeurons, which aggregates lesion‑specific anatomical and phenotypic evidence in a learned latent space. Experiments on LIDC‑IDRI, CBIS‑DDSM, and 3DReasonKnee show that PureVision enhances lesion grounding and phenotype characterization in visual question answering and radiology report generation.

By Hao Wang, Qiwei Zeng, Shuchang Ye, Jinghao Lin, Yuezhe Yang, Yige Peng, Jinman Kim, Lei Bi
arXiv AI
Aug 28

From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation

The paper introduces MedREAL, a unified framework that aligns linguistic reasoning with spatial grounding for medical visual question answering and segmentation. MedREAL employs Seg Anchored Reasoning Pooling (SARP) to extract semantic evidence from segmentation tokens and a Reasoning-to-Visual (R2V) fusion mechanism to integrate these features into a segmentation pipeline. Using the newly created MedRAVS-13K dataset, MedREAL achieves superior performance, reporting 68.49% gIoU and 70.47% cIoU, and generates evidence masks that consistently match textual diagnoses.

By Haowen Gu, Gensheng Pei, Junzhu Mao, Qiong Wang, Mingwu Ren, Yazhou Yao