Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,624 stories · RSS feed

arXiv Computer Vision
Aug 21

MUST-PET: MUltimodal Self-supervised learning across Tracers for whole-body PET/CT-based lesion segmentation

arXiv:2608. 19666v1 Announce Type: new Abstract: Deep learning-based whole-body PET-CT lesion segmentation can support cancer staging, treatment planning, and response assessment, but generalization is limited by scarce annotations and domain shifts.

By Bashirul Azam Biswas, Amartya Bhattacharya, Biratal Raj Wagle, Matthew E. Maeder, James B. Yu, Indrani Bhattacharya
arXiv Computer Vision
Aug 21

ID-VTG: Image-Disambiguated Video Temporal Grounding

arXiv:2608. 20127v1 Announce Type: new Abstract: Video Temporal Grounding (VTG) faces significant challenges when natural language queries must distinguish between multiple events involving visually similar entities, particularly when relying on fine-grained visual attributes that are difficult to describe accurately in words alone.

By Minghang Zheng, Jingli Wei, Hongyi Yang, Yang Liu
arXiv Computer Vision
Aug 21

MOSAIC: Modality-agnostic Spectral Alignment for Federated Image-level Weakly Supervised Tumor Segmentation under Client-specific Missing Modalities

arXiv:2608. 19788v1 Announce Type: cross Abstract: Trustworthy multimodal fusion in clinical settings requires handling incomplete and heterogeneous modality subsets across institutions, where privacy constraints prohibit centralized data sharing.

By Tarun Kumar Garg, Vaanathi Sundaresan
arXiv AI
Aug 21

Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agents

arXiv:2608. 19794v1 Announce Type: new Abstract: The convergence of large language models (LLMs), structured knowledge bases (KBs), and reasoning ability (RA) presents a promising trajectory toward general embodied intelligence (GEI).

By Fujiang Yuan, Xia Huang, Lusheng Wang, Jun Ding, Zhen Tian, Yuxin Wang, Shaojie Gu, Yuki Funabora, Yanhong Peng, Zebing Mao
arXiv Computer Vision
Aug 21

Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation

arXiv:2607. 14497v2 Announce Type: replace Abstract: Egocentric Visual Question Answering (VQA) has attracted widespread attention as an important task for enabling Multimodal Large Language Models (MLLMs) to interact with the real world.

By Chi Kit Wong, Ye Pan, Yuanhuiyi Lyu, Xu Zheng, Zidong Cao, Lutao Jiang, Zixin Zhang, Huiyu Zhou, Xuming Hu
arXiv AI
Aug 21

MKG-CARE: Case-Aware Reasoning with Multimodal Knowledge Graphs for Explainable Medical Image Diagnosis

arXiv:2605. 22547v3 Announce Type: replace-cross Abstract: Medical image diagnosis has achieved significant progress with deep learning, yet existing methods often rely on isolated visual evidence and lack the ability to effectively leverage similar cases and external knowledge.

By Yiming Xu, Yixuan Liu, Yuhang Zhang, Ling Zheng, Yihan Wang, Qi Song