The article surveys brain‑to‑language decoding, covering tasks, neural signals, methods, and evaluation across invasive and non‑invasive modalities. It traces the field’s evolution from constrained recognition to text generation, streaming speech, and facial animation, linking tasks to neural populations and decoder representations. The review highlights complementary decoding targets, shared representations, and the growing importance of calibration, feedback, and user control for online communication, while proposing a five‑level future trajectory toward bidirectional cognitive exchange.
By Yiqian Yang, Yiqun Duan, Chenyu Liu, Yiqi Wang, Xinliang Zhou, Chin-Teng Lin, Yu Zhang
arXiv:2609.36736v1 Announce Type: cross
Abstract: Speech brain-computer interfaces (BCIs) aim to restore communication by transforming neural activity related to speech, language, or communicative in...
By Moein Khajehnejad, Forough Habibollahi, Tommaso Boccato, Margarida Sousa, Michal Olak, Francesco Jamal Sheiban, Matteo Ferrante
arXiv:2610.34457v1 Announce Type: cross
Abstract: Speech and language technologies have expanded access to communication, but the support they provide is often fixed. The same linguistic or visual ai...
By Carlos Bruno, Luis Ricardo Santos
The paper introduces Language as Supervision (LaS), a method that aligns neural activity with natural‑language class descriptions via learned projections and uses candidate text prototypes for classification. Evaluated across seven encoder architectures on visual, motor, and speech decoding tasks from scalp and intracranial recordings, LaS boosts speech‑decoding accuracy by up to 17.15 percentage points compared to traditional class‑label supervision. The approach also enables reusable candidate sets, subject‑ and session‑adaptation, and conditional unseen‑frequency decoding in steady‑state visual evoked potentials, with dual‑sided attribution visualizations highlighting both neural and textual contributions to class matching.
By Xuan Wen, Yonghao Song, Yuhong Zhang, Guoliang Li, Tzyy-Ping Jung, Xiaorong Gao
The paper introduces Alignment-Free Text‑Audiobox (Text‑AB), a unified diffusion‑based framework that performs high‑quality voice dubbing and full‑duplex dialogue synthesis without requiring forced alignment. Text‑AB uses a latent diffusion model with DAC‑VAE features, achieving over 10× compression compared to prior EnCodec representations, and learns text‑speech alignment via cross‑attention. The authors pretrain a 3B‑parameter model on 480k hours of monolingual speech and fine‑tune it for cross‑lingual dubbing, full‑duplex dialogue, and emotional dialogue, reporting significant improvements in prosody, voice similarity, naturalness, and emotional expressivity over existing internal systems.
By Sanyuan Chen, Min-Jae Hwang, Sho Inoue, Anna Sun, Bokai Yu, David Kant, Dongmin Hyun, Dorian Desblancs, Gregory Antonovsky, Oleg Repin, Peng-Jen Chen, Xutai Ma, Zehai Tu, Juan Pino, Wei-Ning Hsu
arXiv:2606. 07547v1 Announce Type: cross Abstract: Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs.
By Luoyuan Zhang, Bokai Xu, Junbo Cui, Weiyue Sun, Yingjing Xu, Hanyu Liu, Yuan Yao
The paper investigates whether large models are necessary for speech brain‑computer interfaces (BCIs) and introduces BEST, a two‑stage framework that uses hierarchical intermediate representations and a compact decoder to map neural activity directly to text. BEST achieves low word error rates (6.19% and 1.62%) on the Brain‑to‑Text ’24 and ’25 benchmarks while using only 3.85% of the parameters of comparable methods. The study also highlights that decoder scaling yields non‑monotonic gains, cross‑modal alignment is more critical than model size, and overfitting harms sequence‑level performance.
By Zehui Feng, Weichuan Wang, Xiaohan Chen, Cuntai Guan, Ting Han
arXiv:2608. 20186v1 Announce Type: new Abstract: Non-invasive decoding of inner speech faces a fundamental data problem: a corpus pairing brain activity with a person's spontaneous inner monologue cannot be collected, and the available proxy paradigms (cued repetitive and retrospectively reported generative inner speech) are slow to acquire, poorly time-locked, and subject compliance is unverifiable.
By Ingo Marquardt, Anthilia Alchanat, Priyanka Jain
FireRedAudio is a 9‑billion‑parameter audio language model that separates continuous input representations for audio understanding and speech generation, enabling a single autoregressive LLM to perform tasks such as ASR, zero‑shot TTS, Instruct TTS, and semantic/acoustic speech editing. The model uses a dedicated Audio Encoder for recognition and a RedAE‑based pathway for generation, with the LLM directly generating text or conditioning a flow‑matching DiT to produce acoustic latents. Evaluations show competitive or leading performance in multilingual ASR, content‑accurate zero‑shot TTS, strong instruction following, and significant improvements in speech editing over prior work.
By Feiyu Shen, Fenglong Xie, Junjie Li, Kun Xie, Lei Xie, Xu Tang, Xuelong Geng, Yan Jia, Yao Hu, Yichen Han, Yichen Wu, Ziqi Dai, Junjie Chen, Kai Huang, Manzhen Wei, Yixuan Li
The paper introduces a geometric measure of coarticulation for speech‑driven 3D facial animation, comparing lip‑path length to the shortest route through vowel, consonant, and vowel positions. Using only forced alignment, the measure evaluates four state‑of‑the‑art animation methods, revealing that all produce flatter lip trajectories than captured speech and that some methods lose 15–60% of the fast articulatory component. A pre‑registered viewer study confirms that damping real motion lowers perceived quality while exaggeration is not penalized, and viewers prefer real speech in 73.4% of sentence comparisons.
By Danzel Serrano, Przemyslaw Musialski
arXiv:2610.22499v1 Announce Type: new
Abstract: Human communication is inherently multimodal, conveying intent not only through linguistic and vocal cues but also through visual signals. However, exi...
By Junseok Ahn, Jaehyun Park, Jeongsoo Choi, Joon Son Chung, Ji-Hoon Kim
arXiv:2609.22697v1 Announce Type: new
Abstract: Recently, text-to-speech systems have made significant progress in speech expressiveness and controllability. However, the speaking style of generated...
By Weizhen Bian, Sitong Cheng, Rongxiu Zhong, Jiahao Pan, Liumeng Xue, Boyi Kang, Shilei Zhang, Jinglei Liu, Yue Wang, Junlan Feng, Bei Liu, Wei Xue