arXiv Machine Learning

Transformers for Multimodal Brain State Decoding: Integrating Functional Magnetic Resonance Imaging Data and Medical Metadata

arXiv:2512. 08462v2 Announce Type: replace Abstract: Decoding brain states from functional magnetic resonance imaging (fMRI) data is vital for advancing neuroscience and clinical applications.

arXiv Computer Vision
Sep 14

Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimaging

The paper introduces Neuro‑JEPA, a sparse multimodal foundation model that learns unified representations of brain MRI across T1w, T2w, and FLAIR sequences using a latent predictive objective and a Mixture‑of‑Experts architecture. It was pretrained on over 1.5 million scans from 428,647 studies and systematically evaluates architectural, masking, objective, and sparsity choices for robust multimodal representation learning. Across 47 tasks from three health systems and 12 public datasets, Neuro‑JEPA consistently outperforms a simple CNN baseline, demonstrating its effectiveness for both clinical and research applications.

By Haoxu Huang, Long Chen, Jingyun Chen, Jinu Hyun, James Ryan Loftus, Kara Melmed, Daniel Orringer, Jennifer Frontera, Seena Dehkharghani, Arjun Masurkar, Narges Razavian
arXiv Machine Learning
Jul 30

Challenges and proposed solutions in modeling multimodal medical data: A systematic review

arXiv:2505. 06945v5 Announce Type: replace Abstract: Multimodal data modeling has emerged as a powerful approach in clinical research, enabling the integration of diverse data types such as imaging, genomics, wearable sensors, and electronic health records.

By Maryam Farhadizadeh, Maria Weymann, Michael Bla{\ss}, Johann Kraus, Christopher Gundler, Sebastian Walter, Noah Hempen, Hannah Bast, Harald Binder, Nadine Binder
arXiv Machine Learning
Jul 30

An Attention-Based Framework for Alzheimers Disease Classification Using Resting-State fMRI

arXiv:2607. 26746v1 Announce Type: cross Abstract: Accurate identification of Alzheimers disease (AD) using resting-state functional magnetic resonance imaging (rs-fMRI) remains challenging due to the high dimensionality, noise, and complex inter-regional dependencies inherent in functional brain connectivity, which limit the effectiveness of traditional approaches based on handcrafted connectivity features or conventional machine learning models.

By Harshiddhi Pathak, Gowtham Reddy N, Mrinal Acharya, Manjunatha Mahadevappa
arXiv AI
Jun 12

BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning

arXiv:2604. 27277v3 Announce Type: replace-cross Abstract: Brain MRI underpins a wide range of neuroscientific and clinical applications, yet most learning-based methods remain task-specific and require substantial labeled data.

By Yizhou Wu, Shansong Wang, Yuheng Li, Mojtaba Safari, Mingzhe Hu, Chih-Wei Chang, Harini Veeraraghavan, Xiaofeng Yang
arXiv Machine Learning
Aug 28

Quantitative mapping from conventional MRI using self-supervised physics-guided deep learning: applications to a large-scale, clinically heterogeneous dataset

This study introduces a self‑supervised, physics‑guided deep‑learning framework that converts standard clinical T1‑, T2‑, and FLAIR MRIs into quantitative T1, T2, and proton‑density maps. Trained on 4,121 scan sessions from four different 3 T scanners over six years, the method produces maps whose white‑ and gray‑matter values fall within literature ranges and shows minimal variation across scanner hardware and acquisition protocols (coefficients of variation ≤ 1.1 %). Voxel‑wise reproducibility is high, with Pearson and concordance correlation coefficients above 0.82 for T1 and T2 and mean relative differences below 6 % for T2.

By Jelmer van Lune, Stefano Mandija, Oscar van der Heide, Matteo Maspero, Martin B. Schilder, Jan Willem Dankbaar, Cornelis A. T. van den Berg, Alessandro Sbrizzi
arXiv AI
Sep 12

Exploring Diffusion Transformers for Cross-Modal Augmentation in Multimodal Brain State Decoding

The paper introduces CoMA-DiT, a bidirectional cross‑modal Diffusion Transformer that uses paired modalities as mutual generative supervision for latent augmentation rather than just inputs for fusion. By conditioning velocity prediction on the paired modality through cross‑modal attention and injecting variation via a reliability‑gated residual mechanism, CoMA‑DiT improves multimodal brain state decoding. Experiments on auditory attention decoding and emotion recognition show consistent gains over 20 baselines, with absolute accuracy and macro‑F1 improvements of 4.28% and 6.70% respectively, and extensive analyses confirm its robustness and interpretability.

By Ziwei Wang, Xingyi He, Hongbin Wang, Tianwang Jia, Bohan Fang, Dongrui Wu