arXiv Computer Vision

Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction

arXiv Computer Vision
Sep 14

Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimaging

The paper introduces Neuro‑JEPA, a sparse multimodal foundation model that learns unified representations of brain MRI across T1w, T2w, and FLAIR sequences using a latent predictive objective and a Mixture‑of‑Experts architecture. It was pretrained on over 1.5 million scans from 428,647 studies and systematically evaluates architectural, masking, objective, and sparsity choices for robust multimodal representation learning. Across 47 tasks from three health systems and 12 public datasets, Neuro‑JEPA consistently outperforms a simple CNN baseline, demonstrating its effectiveness for both clinical and research applications.

By Haoxu Huang, Long Chen, Jingyun Chen, Jinu Hyun, James Ryan Loftus, Kara Melmed, Daniel Orringer, Jennifer Frontera, Seena Dehkharghani, Arjun Masurkar, Narges Razavian
arXiv Machine Learning
Sep 24

A Scaling Study for fMRI Foundation Models

The study investigates how data volume, model size, and training duration affect the performance of fMRI foundation models. Using over 200 datasets and 10,000 GPU‑hours, the authors find that larger models benefit more from additional data, and that at a fixed compute budget, increasing data yields greater gains than enlarging the model. By selecting optimal combinations of data, size, and duration, they produce models that outperform existing fMRI foundation models on out‑of‑distribution tasks while requiring less pretraining compute.

By Wenhao Ye, Xuanye Pan, Junfeng Xia, Junxiang Zhang, Mo Wang, Quanying Liu
arXiv Computer Vision
Oct 16

FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps

FlatClip is a method that leverages frozen image foundation models to predict fMRI activity by mapping cortical signals onto anatomy‑linked flatmaps and extracting features with a SigLIP2 encoder. Only a lightweight downstream probe is trained, avoiding fMRI‑specific encoder pretraining or fine‑tuning. Experiments on four resting‑state benchmarks show FlatClip surpasses ROI‑level baselines on HCP and ADNI, while its performance varies on PPMI and remains below the best voxel‑level models; on NSD visual‑fMRI decoding, restricting inputs to task‑active cortex improves recognition, and spatial perturbations confirm the importance of cortical arrangement and input‑region selection.

By Mo Wang, Wenhao Ye, Zihan Ning, Jiayu Zuo, Junfeng Xia, Hongkai Wen, Quanying Liu
arXiv AI
Sep 21

Rhamba: Region-Aware Hybrid Attention-Mamba Framework for Self-Supervised Learning in Resting-State fMRI

Rhamba is a region‑aware pretraining framework for resting‑state fMRI that combines anatomically guided masking with hybrid Attention‑Mamba architectures. The study pretrained models on the ABIDE dataset using three masking strategies (Any, Majority, Pure) and evaluated four architectural variants, finding that the Mamba‑Attention (MA) hybrid achieved the best average AUROC on downstream schizophrenia and ADHD classification tasks. Explainable AI via Integrated Gradients highlighted that performance depends on the interaction between masking strategy and architecture rather than a single dominant configuration.

By Pankaj Pandey, Ruthwik Reddy Doodipala, Pratheek Eranki, Carolina Torres-Rojas, Manob Jyoti Saikia, Ranganatha Sitaram
arXiv AI
Oct 31

Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI

The study investigates whether self‑supervised learning can produce useful representations from fMRI data alone. Using a joint embedding predictive architecture, participant‑specific encoders are trained without access to stimulus images or derived representations. The resulting embeddings perform competitively on brain captioning, image reconstruction, and cross‑participant generalization, and can be aligned unsupervised across participants, enabling a decoding head trained on one subject to work on another with minimal loss of performance.

By Serwan Jassim, Victoria Bosch, Daniel Anthes, Elia Bruni, Tim Kietzmann
arXiv Computer Vision
Oct 12

Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability

Neurons++ is a visual‑stream framework that reconstructs both static images and dynamic videos from fMRI data. It builds on the earlier Neurons model by extending from video‑only to image‑and‑video reconstruction and replacing parallel task conditioning with a serial, cortex‑inspired decoding pipeline. The approach first pre‑trains on image‑fMRI pairs for stable spatial‑semantic initialization, then fine‑tunes separately for image and video tasks, organizing fMRI representations into spatial, semantic, language‑level, and temporal cues, and achieves state‑of‑the‑art video reconstruction with competitive image results while enabling neural‑to‑visual interpretability and ROI‑guided semantic modulation.

By Haonan Wang, Yuchen Wang, Jingyu Lu, Qixiang Zhang, Lehan Wang, Xuanqi Huang, Xiaomeng Li