The paper investigates how Spoken Language Models (SLMs) process speech compared to text, noting that current SLMs show weak alignment between speech and text representations despite strong downstream performance. The authors propose a framework that separates length mismatch from semantic alignment to better match speech and text representations. Experiments on multiple benchmarks demonstrate that this approach yields competitive results against strong baselines, highlighting the need to explicitly address structural differences between speech and text in SLM training.
By Hyeonyu Kim, Hwayeon Kim, Youngwon Choi, Myeongkyun Cho, Huu-Kim Nguyen
SPAR-K is a scheduled periodic alternating early‑exit framework for interleaved spoken language models that reduces decoding depth for speech tokens while maintaining quality. It lets most speech positions exit at a fixed intermediate layer and inserts periodic full‑depth refresh steps to counter distribution shift. Experiments on Step‑Audio‑2‑mini and GLM‑4‑Voice show up to 11 % depth reduction with less than 0.82 % drop in question‑answering accuracy and negligible impact on MOS and WER.
By Hsiao-Ying Huang, Cheng-Han Chiang, Hung-yi Lee
Connecting a pre-trained speech encoder to a Large Language Model (LLM) is the standard architecture for building Speech LLMs. However, a structural misalignment exists between the encoder and the LLM.
Uni-Stream is a unified streaming speech language model that supports both speech recognition and synthesis within a single architecture. Built on compact pretrained speech models, it processes interleaved text and speech in real time and achieves performance comparable to task‑specific models through a two‑stage training approach. The authors also introduce a data annotation pipeline for streaming training and plan to release the generated transcriptions and word‑level timestamps, while analyzing how speech and text representations occupy distinct latent spaces in the model.
By Xuan Shi, Chang Zeng, Tiantian Feng, Ye Zhang, Jun Liu, Shih-Heng Wang, Lie Lu, Shrikanth Narayanan
arXiv:2609.15313v1 Announce Type: cross
Abstract: Autoregressive generation of interleaved text and acoustic tokens is a common approach to spoken-response generation in speech large language models....
By Daxin Tan, Dehua Tao, Chengxi Deng, Hanlin Zhang, Xiao Chen
arXiv:2506. 16738v2 Announce Type: replace-cross Abstract: With the rapid progress of speech language models (SLMs), discrete speech tokens have emerged as a core interface between speech and text, enabling unified modeling across modalities.
By Daejin Jo, Jeeyoung Yun, Byungseok Roh, Sungwoong Kim
arXiv:2606. 10029v1 Announce Type: cross Abstract: Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tokens share a single residual stream.
By Nikita Koriagin, Georgii Aparin, Nikita Balagansky, Daniil Gavrilov
arXiv:2607.01733v2 Announce Type: replace
Abstract: Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech reco...
By Ruchao Fan, Yiming Wang, Rui Zhao, Liliang Ren, Keqi Deng, Xiaoyang Chen, Ali Zare, Bo Ren, Yuxuan Hu, Junkun Chen, Yan Huang, Yelong Shen, Jinyu Li
The study investigates how augmenting training data with conceptually related alternatives—beyond nouns to include verbs and adjectives—affects large language models trained with next-token prediction. Results show that while concept augmentation consistently improves perplexity on news data regardless of part of speech, verbs yield the most reliable downstream performance gains. Representational analyses reveal that the conditions producing the largest downstream improvements do not always correspond to the greatest changes in internal representations.
By Laya Iyer, Chen Shani
arXiv:2609.10434v1 Announce Type: new
Abstract: Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the b...
By Robin Huo, Ewan Dunbar
arXiv:2603.26246v2 Announce Type: replace-cross
Abstract: Standard LLM-based speech recognition systems process utterances in isolation, limiting their ability to leverage conversational context. Her...
By Shashi Kumar, Esa\'u Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Ba\~neras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke
The paper introduces Hybrid Search, a method that refines warm-initialized large language model (LLM) based automatic speech recognition (ASR) systems by exploiting interactions between ASR hidden states and the base LLM’s hidden states. By identifying tokens with high semantic dependence and selectively correcting them, the approach surpasses traditional global LLM‑correction techniques such as rescoring and late fusion. The study demonstrates that even after warm initialization, LLM‑based ASR models can further benefit from their base LLM during inference.
By Chan-Jan Hsu, Jaeyeon Kim, Chao-Han Huck Yang, Shinji Watanabe, Hung-yi Lee, Carlos Busso