arXiv Machine Learning August 19, 2026 By Paul Minchella, St\'ephane Chr\'etien, Guillaume Metzler, Lo\"ic Verlingue, R\'emi Vaucher

MultiSigBERT: Beyond Survival Analysis through Multimodal and Sequential Modeling in Oncology

Read the original on arXiv Machine Learning →

MultiSigBERT is a unified framework that performs multimodal sequential survival modeling in oncology by integrating narrative clinical reports, numerical measurements, and structured variables. The method converts free-text reports into sentence embeddings, compresses them with modality-specific PCA, and concatenates them with structured covariates to create joint temporal trajectories. These trajectories are encoded using the Signature transform from Rough Paths theory, and the resulting high-dimensional features are fed into a LASSO-regularized Cox model, achieving a concordance index of 0.743 on an independent test set of over 2,500 patients.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Aug 4

EHR2Path: Comprehensive Pathway-Level Modeling of Longitudinal Patient Trajectories from Multimodal Electronic Health Records

arXiv:2506. 04831v3 Announce Type: replace Abstract: Forecasting how a patient's condition is likely to evolve, including possible deterioration, recovery, treatment needs, and care transitions, could support more proactive and personalized care, but requires modeling heterogeneous and longitudinal electronic health record (EHR) data.

By Chantal Pellegrini, Ege \"Ozsoy, David Bani-Harouni, Matthias Keicher, Nassir Navab

nlp multimodal

arXiv Machine Learning

Jul 14

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

arXiv:2607. 09982v1 Announce Type: new Abstract: Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance.

By Nikkie Hooman, Zhongjie Wu, Eric C. Larson, Mehak Gupta

multimodal safety

arXiv AI

Jun 30

Primary ICD Category Prediction using LLM-based Probing

arXiv:2606. 28798v1 Announce Type: new Abstract: Objective: ICD codes are central to reimbursement, research, and population health surveillance, yet automated coding systems often struggle to integrate diagnostic signals from both clinical narratives and structured electronic health record (EHR) variables.

By Chengyuan Liu, Xinyue Zhang, Yao Li, Guanting Chen

llms rag multimodal

arXiv AI

Jul 20

Large Language Models as Unified Multimodal Learners for Clinical Prediction

arXiv:2607. 15380v1 Announce Type: cross Abstract: Electronic health records combine free-text clinical narratives with structured measurements such as vital signs, laboratory values, and comorbidities.

By Ajay Madhavan Ravichandran, Bilgin Osmandoja, Klemens Budde, Klaus Netter, Tobias Strapatsas, Aljoscha Burchardt, Sebastian M\"oller, Roland Roller

llms fine-tuning multimodal

arXiv Machine Learning

Aug 26

A Multimodal Foundation Model for Longitudinal Patient Representation and Scalable Insight Generation in Oncology

arXiv:2608.24688v1 Announce Type: new Abstract: Precision oncology necessitates a longitudinal model of patient state that captures cancer evolution and treatment over time, integrating multimodal ob...

By Eugene Vorontsov, Yi Kan Wang, Alican Bozkurt, Adam Casson, Ludmila Tydlitatova, Michal Zelechowski, Ezra E. W. Cohen, Jyoti D. Patel, Max Banaszak, Caitlin McWilliams, Shane Colley, Kate Sasser, Ryan Fukushima, Eric Lefkofsky, Razik Yousfi, Siqi Liu

rag multimodal benchmarks

arXiv Computer Vision

Sep 7

Real-World Multi-Modal and Longitudinal Lung Cancer Dataset

The paper presents a newly curated, multi-center, multi-modal, and longitudinal lung cancer dataset comprising 1,365 patients with whole-slide images, CT scans, PET scans, structured clinical data, transcriptomics, and follow-up information. The dataset features substantial, non-uniform missingness across modalities, making it ideal for evaluating robust multi-modal fusion strategies. Benchmarks on 12‑month overall survival, disease‑specific survival, and longitudinal hazard prediction demonstrate that integrating complementary modalities consistently outperforms uni-modal approaches, even under severe missing data.

By Rita Cordeiro Mendes, Maria Rita Fonseca Verdelho, Carlos Santiago, Catarina Barata

multimodal benchmarks

One email a morning, machine-written

Related stories

EHR2Path: Comprehensive Pathway-Level Modeling of Longitudinal Patient Trajectories from Multimodal Electronic Health Records

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

Primary ICD Category Prediction using LLM-based Probing

Large Language Models as Unified Multimodal Learners for Clinical Prediction

A Multimodal Foundation Model for Longitudinal Patient Representation and Scalable Insight Generation in Oncology

Real-World Multi-Modal and Longitudinal Lung Cancer Dataset