arXiv AI

OA-MAP: Evidence-Grounded Multi-Agent Multimodal Framework for Interpretable Knee Osteoarthritis Progression

OA-MAP is an autonomous multi‑agent multimodal framework designed to predict knee osteoarthritis progression by integrating MRI, X‑ray, and clinical data. The system employs modality‑specific agents and a coordinator to autonomously select predictive tools, retrieve supporting literature, and provide interpretable risk estimates. In a study of 600 participants, the fusion models achieved AUROCs of 0.80 for structural progression and 0.68 for pain progression, and a case study demonstrated how OA‑MAP combines risk estimates, intermediate findings, literature support, and uncertainty indicators for interactive clinician review.

arXiv AI

Jun 6

An interpretable and trustworthy AI framework for large-scale longitudinal structure-pain association studies using data from the Osteoarthritis Initiative (OAI)

arXiv:2606. 05357v1 Announce Type: new Abstract: Purpose: To develop an interpretable and trustworthy AI framework that combines deep learning based MRI Osteoarthritis Knee Score (MOAKS) prediction with interpretable statistical modeling to study structure-pain relationships at scale using data from the Osteoarthritis Initiative (OAI).

By Jincheng Yu, Haoyang Li, Yiwen Liu, Shen Liu, Rachel Yuanbao Chen, C. Kent Kwoh, Hongxu Ding, Xiaoxiao Sun

arXiv AI

Oct 7

Knee3DVLM: Dual-Sequence Full-Volume Vision-Language Modeling for Comprehensive Knee MRI Assessment

arXiv:2610.08482v1 Announce Type: cross Abstract: Vision-language models (VLMs) are increasingly being applied to three-dimensional medical imaging, but their application to knee MRI remains limited,...

By Maryam Baizhigitova, Andrew Seohwan Yu, Po-Hao Chen, Naveen Subhas, Sixu Chen, Xinxin Wang, Kunio Nakamura, Richard Lartey, Xiaojuan Li, Mingrui Yang

llms multimodal

arXiv AI

Jun 9

A multi-agent system for spine MRI report generation from multi-sequence imaging

arXiv:2606. 08897v1 Announce Type: cross Abstract: Spinal pathology is a leading cause of pain and disability worldwide.

By Zhiping Xiao, Junwei Yang, Gongbo Sun, Han Zhang, Hanwen Xu, Yi Yao, Zachary D. Miller, William E. King III, Mohammed M. Kanani, Jalal B. Andre, Sammy Chu, Ming Zhang, Paul E. Kinahan, Nathan M. Cross, Sheng Wang

rag agents multimodal benchmarks

arXiv AI

Jun 17

Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis

arXiv:2606. 17115v1 Announce Type: cross Abstract: Foundation models (FMs) have emerged as powerful representation extractors for medical data, yet their generalizability to datasets under distribution shift remains underexplored.

By Jingyu Hu, Giuseppe Tripodi, Reed Naidoo, Sarah F. McGough, Tapabrata Chakraborti

multimodal benchmarks

arXiv AI

Jun 9

Knee-xRAI: An Explainable AI Framework for Automatic Kellgren-Lawrence Grading of Knee Osteoarthritis

arXiv:2604. 23435v2 Announce Type: replace-cross Abstract: Grading knee osteoarthritis (KOA) on plain radiographs is poorly reproducible across readers.

By Azmul A. Irfan, Nur Ahmad Khatim, Alfan Alfian Irfan, Achmad Zaki, Erike A. Suwarsono, Mansur M. Arief

computer-vision

arXiv Computer Vision

Aug 24

Kinematic Knowledge Maps for Pattern Alignment: Structured Latent Representational Learning in Multimodal Gait Analysis

arXiv:2608.20969v1 Announce Type: new Abstract: Multimodal clinical AI is limited by weakly aligned inputs and the absence of domain-specific interpretable representations, particularly when learning...

By Chen Dong, He Zonglin, Cheung Kenneth M. C

rag multimodal safety

arXiv AI

2d ago

Multimodal Fusion Methods for Whole-Slide Images and Clinical Data in Inflammatory Bowel Disease

The paper introduces a modular framework for multimodal AI fusion that combines slide-level embeddings from histology images with per-visit clinical embeddings using 15 interchangeable fusion methods. Applied to a prospective cohort of newly diagnosed inflammatory bowel disease patients, the best fusion method achieved macro‑F1 scores between 0.57 and 0.75, outperforming single‑modality models in most scenarios and showing varying reliance on image data depending on the remission outcome. The study highlights how different fusion strategies can influence predictive performance across distinct clinical endpoints.

By Ankana Banerjee (University of Stavanger, Norway), Mark van der Giezen (University of Stavanger, Norway), Ole Gunnar Aasprong (Stavanger University Hospital, Norway), Emiel A. M. Janssen (University of Stavanger, Norway), {\O}yvind Steinsb{\o} (Stavanger University Hospital, Norway), Tore Grimstad (Stavanger University Hospital, Norway), Kjersti Engan (University of Stavanger, Norway)

rag multimodal

arXiv Computer Vision

Oct 27

Saga: an in silico ontology for a multimodal patient world model

arXiv:2610.33054v1 Announce Type: new Abstract: Simulating an individual's disease course requires connecting patient history to future events, clinical timing and disease morphology. Longitudinal pr...

By Youcheng Li (Peking University, Isoplex Lab), Fa Tian (Isoplex Lab), Binbin Su (Peking Union Medical College, Isoplex Lab)

llms multimodal

arXiv Machine Learning

Jul 14

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

arXiv:2607. 09982v1 Announce Type: new Abstract: Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance.

By Nikkie Hooman, Zhongjie Wu, Eric C. Larson, Mehak Gupta

multimodal safety

arXiv AI

Sep 1

FRAC-MAS: A Safe and Explainable Multi-Agent System for Fracture Diagnosis

arXiv:2608.28662v1 Announce Type: new Abstract: Fracture detection and its clinical interpretability see notable improvements when deep vision models are integrated with agentic AI architectures. Whi...

By Hardik Iyer, Tirath Bhathawala, Mihir Panchal, Ying-Jung Chen, Kiran Bhowmick, Pankaj Sonawane, Meera Narvekar

llms agents benchmarks safety

arXiv Computation and Language

Sep 15

MedTRACE: Tool-Augmented Multimodal Clinical Reasoning Agents for Evidence-Grounded Decision-Making

arXiv:2609.14823v1 Announce Type: new Abstract: Multimodal clinical decision-making requires reliable reasoning over heterogeneous evidence from electronic health records, medical images, and physiol...

By Ji Lu, Lifei Liu, Haoran Yu, Xianglong Wang, Yiru Fang, Kuo Yang, Huiran Duan, Jianping Gou

agents multimodal benchmarks safety

arXiv AI

Aug 19

A Multimodal Agentic Pathology Co-pilot via Evidence Grounded Reasoning

PathPocket is a multimodal AI co‑pilot that grounds pathology decision‑making in evidence. It builds the largest pathology evidence corpus (≈110,472 documents) and a hypergraph of 4.55 million entities and 7.10 million relations to support traceable reasoning. The system handles text and multimodal queries, including ROI and gigapixel whole‑slide images, and outperforms current state‑of‑the‑art models on a benchmark of over 200,000 real‑world cases, improving pathologists’ diagnostic accuracy and confidence.

By Zhe Xu, Zhengyu Zhang, Zhiyuan Cai, Jiahao Xu, Yijie Lin, Ziyi Liu, Junlin Hou, Hongyi Wang, Yuxiang Nie, Yihui Wang, Jiabo Ma, Ling Liang, Yingxue Xu, Zhengrui Guo, Guanghao Wu, Danyi Li, Ziqi Zhou, Donglin Tan, Zhijian Cen, Ying Tan, Xiaolin Liu, Qi Xie, Xiaoying Tang, Xi Peng, Cheng Deng, Lijuan Qu, Ronald Cheong Kin Chan, Li Liang, Hao Chen

agents multimodal benchmarks

Related stories