arXiv Computer Vision

Hierarchical MoE for Multi-Modal ILD Diagnosis

The paper introduces a hierarchical multimodal mixture-of-experts (MoE) model for interstitial lung disease (ILD) classification. It combines a frozen, pre‑trained imaging expert with structured electronic health records (EHR) through a two‑stage gating system: a modality‑level gate weights imaging and EHR predictions, while a sub‑gating module further decomposes the EHR branch into clinically defined feature groups with learned, group‑specific contributions. The approach preserves stable imaging representations, allows input‑dependent clinical weighting, and enhances interpretability across anatomical regions, imaging–EHR utilization, and EHR feature groups, achieving the highest mean AUC (0.8750 ± 0.0443) under strict patient‑level cross‑validation.

arXiv AI

Sep 17

Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging

The paper introduces Generalist‑Specialist Mixture‑of‑Experts (GS‑MoE), a two‑branch architecture that combines a cross‑modal generalist model with modality‑specific specialists through domain‑constrained feature fusion. GS‑MoE improves detection of rare pathologies in multimodal medical imaging, achieving significant per‑class F1 gains and outperforming dense and specialist‑only MoE baselines while using about 53% fewer active parameters at inference. The study demonstrates that balancing cross‑modal shared representations with expert routing can enhance performance on low‑prevalence conditions.

By Johannes Kaiser, Florian Braunmiller, Daniel R\"uckert, Georgios Kaissis

multimodal

arXiv AI

Sep 23

A General Multimodal Probing Framework for Data Fusion and Prediction with Frozen Large Language Models: Evidence from Multimodal EHR Data

arXiv:2606.28798v2 Announce Type: replace Abstract: The main objective of this paper is to propose a general framework for prediction based on different sources of multimodal data in the healthcare d...

By Chengyuan Liu, Xinyue Zhang, Yao Li, Guanting Chen

llms rag fine-tuning multimodal

arXiv AI

Jun 30

Primary ICD Category Prediction using LLM-based Probing

arXiv:2606. 28798v1 Announce Type: new Abstract: Objective: ICD codes are central to reimbursement, research, and population health surveillance, yet automated coding systems often struggle to integrate diagnostic signals from both clinical narratives and structured electronic health record (EHR) variables.

By Chengyuan Liu, Xinyue Zhang, Yao Li, Guanting Chen

llms rag multimodal

arXiv AI

Jul 9

ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities

arXiv:2607. 06633v1 Announce Type: cross Abstract: In this paper, we address the problem of multimodal federated learning with missing modality.

By Aavash Chhetri, Bibek Niroula, Eduard Vazquez, Yash Raj Shrestha, Prashnna Gyawali, Loris Bazzani, Binod Bhattarai

multimodal benchmarks

arXiv Machine Learning

Aug 3

MoPET: Parameter-Efficient Mixture-of-Experts for Unified Medical Image Classification

arXiv:2607. 29462v1 Announce Type: cross Abstract: Adapting deep learning models to profound clinical heterogeneity typically relies on parameter-efficient fine-tuning (PEFT) to avoid the severe overfitting associated with full end-to-end network updates.

By Sebastian Doerrich, Daniel W\"urtinger, Francesco Di Salvo, Shyam Nandan Rai, Christian Ledig

computer-vision fine-tuning benchmarks

arXiv Machine Learning

Jul 14

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

arXiv:2607. 09982v1 Announce Type: new Abstract: Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance.

By Nikkie Hooman, Zhongjie Wu, Eric C. Larson, Mehak Gupta

multimodal safety

arXiv Computer Vision

Sep 7

Real-World Multi-Modal and Longitudinal Lung Cancer Dataset

The paper presents a newly curated, multi-center, multi-modal, and longitudinal lung cancer dataset comprising 1,365 patients with whole-slide images, CT scans, PET scans, structured clinical data, transcriptomics, and follow-up information. The dataset features substantial, non-uniform missingness across modalities, making it ideal for evaluating robust multi-modal fusion strategies. Benchmarks on 12‑month overall survival, disease‑specific survival, and longitudinal hazard prediction demonstrate that integrating complementary modalities consistently outperforms uni-modal approaches, even under severe missing data.

By Rita Cordeiro Mendes, Maria Rita Fonseca Verdelho, Carlos Santiago, Catarina Barata

multimodal benchmarks

arXiv Machine Learning

Jul 10

Benchmark Evaluation of Feredated Learning on Multi-organ Images

arXiv:2607. 08219v1 Announce Type: cross Abstract: The privacy requirements of medical data and its substantial variations across organs and modalities hinder the clinical implementation of medical AI.

By Junbin Mao, Xu Tian, Jianchun Zhu, Ludi Li, Jin Liu

benchmarks safety

arXiv Machine Learning

Aug 7

Benchmark Evaluation of Federated Learning on Multi-organ Images

arXiv:2607. 08219v2 Announce Type: replace-cross Abstract: The privacy requirements of medical data and its substantial variations across organs and modalities hinder the clinical implementation of medical AI.

By Junbin Mao, Xu Tian, Jianchun Zhu, Ludi Li, Jin Liu

benchmarks safety

arXiv Computer Vision

Sep 1

Whole-Body MRI Classification via Prompt-Based Clinical Conditioning

arXiv:2608.30824v1 Announce Type: new Abstract: Combining whole-body magnetic resonance imaging (WB-MRI) with clinical variables has the potential to improve systemic disease diagnosis by leveraging...

By Laura Daza, Marta Hasny, Cristina Gonz\'alez, Julia A. Schnabel

llms computer-vision multimodal

arXiv Machine Learning

Jul 31

TIER-MoE: Trust-Informed Expert Routing via Conditional Modality Risk for Multimodal Fusion in Biomedical Classification

arXiv:2607. 27289v1 Announce Type: new Abstract: The promise of multimodal fusion lies in combining complementary sources of evidence, yet more evidence does not always yield a better prediction.

By Yu Chang, Anzhe Cheng, Chenwei Wu, Zhuoran Wang, Jiahao Chen, Tamoghna Chattopadhyay, Sophia I. Thomopoulos, Paul M. Thompson, Liyue Shen, Paul Bogdan

multimodal benchmarks

arXiv Machine Learning

Aug 17

MedMix: Specialization-Consistent Federated Sparse MoEs under Modality Heterogeneity

arXiv:2608. 13911v1 Announce Type: new Abstract: Federated multimodal medical AI faces modality heterogeneity at both the client and sample levels: clients may systematically lack access to specific modality types, while individual records within the same client may contain different partial modality subsets.

By Adiba Orzikulova, Dong Min Kim, Jaehong Yoon, Sung-Ju Lee

multimodal safety

Related stories