← Back to all news
arXiv AI

Teaching neural decoders with language

The paper introduces Language as Supervision (LaS), a method that aligns neural activity with natural‑language class descriptions via learned projections and uses candidate text prototypes for classification. Evaluated across seven encoder architectures on visual, motor, and speech decoding tasks from scalp and intracranial recordings, LaS boosts speech‑decoding accuracy by up to 17.15 percentage points compared to traditional class‑label supervision. The approach also enables reusable candidate sets, subject‑ and session‑adaptation, and conditional unseen‑frequency decoding in steady‑state visual evoked potentials, with dual‑sided attribution visualizations highlighting both neural and textual contributions to class matching.

Related stories

arXiv Computation and Language
Sep 24

Brain-to-Language Decoding: Tasks, Signals, Methods, Evaluation, Practical Use and Beyond

The article surveys brain‑to‑language decoding, covering tasks, neural signals, methods, and evaluation across invasive and non‑invasive modalities. It traces the field’s evolution from constrained recognition to text generation, streaming speech, and facial animation, linking tasks to neural populations and decoder representations. The review highlights complementary decoding targets, shared representations, and the growing importance of calibration, feedback, and user control for online communication, while proposing a five‑level future trajectory toward bidirectional cognitive exchange.

By Yiqian Yang, Yiqun Duan, Chenyu Liu, Yiqi Wang, Xinliang Zhou, Chin-Teng Lin, Yu Zhang
benchmarks
More like this →
arXiv Machine Learning
5d ago

EEG-Captioner: Grounding EEG-to-Language Generation in Fundamental Signal Properties

EEG-Captioner introduces a new EEG-to-language framework that grounds language generation in measurable EEG signal properties. The authors built a large dataset of 160,781 EEG segments annotated with question‑answer pairs targeting spectral, spatial, temporal, and lateral properties, and paired each segment with a feature‑grounded caption. Their Dual‑Domain EEG Encoder jointly models temporal waveforms and spectral‑spatial representations, achieving a macro‑averaged balanced accuracy of 88.81% on signal‑level QA and improving caption factual consistency from 82.06% to 83.46% compared to caption‑only models.

By Leyuan Jin, Jiahao Mei, Yixuan Yuan, Shimeng Li, Mengyue Wu, Yue Ding
llmsnlp
More like this →
arXiv AI
Aug 25

Cross-Subject Generalization in Decoding Perceived Speech from Non-Invasive Brain Recordings

The paper introduces a Cross-Subject Perceived Speech Decoding (CPSD) framework that tackles the challenge of decoding perceived speech from non‑invasive brain recordings across different subjects. CPSD uses a two‑stage training process: first, contrastive learning pre‑trains a source model on multiple subjects to capture shared representations; second, personal specialization fine‑tunes the model for a target subject by extracting consistent components and further training on that subject’s data. A Positional Encoding‑based Spatial Attention (PESA) module is added to remap MEG/EEG data into a standardized reference space, improving cross‑subject consistency. Evaluations on three datasets (Armeni 2022, PKUEEG 2025, Broderick 2018) show that CPSD outperforms baseline methods by more than 6.8%, 15.4%, and 15.8% in Top‑10 accuracy, demonstrating its effectiveness, efficiency, and robustness.

By Aoke Zhang, Bo Wang, Xihong Wu, Heping Cheng, Jing Chen
fine-tuning
More like this →
arXiv AI
Aug 20

Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings

Brain2Qwerty v2 is a model that decodes natural sentences from real‑time magnetoencephalography (MEG) recordings, achieving an average word error rate of 39% across 22,000 sentences typed by nine subjects. The model uses character, word, and sentence‑level representations and shows that decoding accuracy improves log‑linearly with more data, narrowing the gap to intracranial brain‑computer interfaces. AI contributes by replacing hand‑crafted event detection with deep learning, fine‑tuning large language models for semantic extraction, and employing AI agents to refine the decoding pipeline through automated code development.

By Mingfang Zhang, Jarod L\'evy, Cedric Rommel, J\'er\'emy Rapin, Corentin Bel, Julie Bonnaire, Daniel Nieto, Pierre Bourdillon, Svetlana Pinet, St\'ephane d'Ascoli, Thomas Moreau, Jean-R\'emi King
llmsagentsfine-tuning
More like this →