← Back to all news
arXiv AI By Corentin Caris, Louis Jalouzot, Nona Rajabi, G\'eraud Ilinca, Ahmed Hassayoune, Asilbek Muminov, Thadd\'ee Delebarre, Clara Fonteneau, Charles Maussion, Bertrand Thirion, Alexis Thual, Antoine Collas

Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment

Read the original on arXiv AI →

The paper introduces NOEMA, a temporal convolutional decoder that aligns consecutive fMRI windows with multimodal embeddings (text, audio, and speech) of naturalistic speech stimuli. By leveraging these complementary representations, NOEMA can retrieve the correct speech segment from a pool of 2,000 candidates in the top 10 for over 42% of held‑out queries, outperforming baseline decoders even when the candidate set is expanded with thousands of unseen distractors. Ablation studies confirm that both temporal modeling and multimodal alignment are key to its performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 5

Cephalonauts One: A deep fMRI dataset for decoding naturalistic speech in the human brain

Cephalonauts One is a 3 Tesla fMRI dataset featuring 30 hours of brain activity recorded from three healthy subjects while they listened to native-language audio podcasts. The release includes the raw fMRI data, corresponding podcast audio, transcript annotations, and derived stimulus embeddings, making it the deepest naturalistic speech fMRI dataset available. A brain‑decoding benchmark is introduced, framing audio segment retrieval as a task where decoders must match fMRI activity to the correct time‑aligned podcast segment, with standardized splits, metrics, and baseline models provided.

By Antoine Collas, Louis Jalouzot, G\'eraud Ilinca, Corentin Caris, Romain Valabr\`egue, Ahmed Hassayoune, David Goncalves, Madeleine Hueber, Thadd\'ee Delebarre, Julien Savatovsky, Clara Fonteneau, Charles Maussion, Bertrand Thirion, Alexis Thual
ragbenchmarks
More like this →
arXiv AI
Sep 28

BAT-CLIP: Trimodal Alignment of Brain, Audio and Text

BAT-CLIP is a trimodal alignment framework that jointly aligns intracranial EEG (iEEG) neural embeddings to both pretrained audio and text anchors within a shared, frozen audio‑text manifold. Unlike existing CLIP‑style brain‑speech models that anchor neural activity to a single modality, BAT‑CLIP leverages both audio and text to preserve temporal structure and linguistic separability. On the naturalistic Podcast benchmark, BAT‑CLIP produces more robust representations than bimodal CLIP baselines and demonstrates the value of self‑supervised foundation models for CLIP training.

By Suhyun Kim, Jinmo Han, Danny Dongyeop Han, Ahhyun Lucy Lee, Jewoon Lee, Yonghyeon Gwon, Zach Paris, Chun Kee Chung, Saewoong Bahk, Nam Soo Kim, Seong Jae Hwang, Jiook Cha
ragmultimodalbenchmarkssafety
More like this →
arXiv AI
Aug 25

Cross-Subject Generalization in Decoding Perceived Speech from Non-Invasive Brain Recordings

The paper introduces a Cross-Subject Perceived Speech Decoding (CPSD) framework that tackles the challenge of decoding perceived speech from non‑invasive brain recordings across different subjects. CPSD uses a two‑stage training process: first, contrastive learning pre‑trains a source model on multiple subjects to capture shared representations; second, personal specialization fine‑tunes the model for a target subject by extracting consistent components and further training on that subject’s data. A Positional Encoding‑based Spatial Attention (PESA) module is added to remap MEG/EEG data into a standardized reference space, improving cross‑subject consistency. Evaluations on three datasets (Armeni 2022, PKUEEG 2025, Broderick 2018) show that CPSD outperforms baseline methods by more than 6.8%, 15.4%, and 15.8% in Top‑10 accuracy, demonstrating its effectiveness, efficiency, and robustness.

By Aoke Zhang, Bo Wang, Xihong Wu, Heping Cheng, Jing Chen
fine-tuning
More like this →