arXiv Machine Learning
Interleaved Speech Language Models Latently Work In Text
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 25
Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text
The paper investigates how Spoken Language Models (SLMs) process speech compared to text, noting that current SLMs show weak alignment between speech and text representations despite strong downstream performance. The authors propose a framework that separates length mismatch from semantic alignment to better match speech and text representations. Experiments on multiple benchmarks demonstrate that this approach yields competitive results against strong baselines, highlighting the need to explicitly address structural differences between speech and text in SLM training.
arXiv Computation and Language
Aug 28
SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Models
SPAR-K is a scheduled periodic alternating early‑exit framework for interleaved spoken language models that reduces decoding depth for speech tokens while maintaining quality. It lets most speech positions exit at a fixed intermediate layer and inserts periodic full‑depth refresh steps to counter distribution shift. Experiments on Step‑Audio‑2‑mini and GLM‑4‑Voice show up to 11 % depth reduction with less than 0.82 % drop in question‑answering accuracy and negligible impact on MOS and WER.
Hugging Face Trending Papers
Jun 24
Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?
Connecting a pre-trained speech encoder to a Large Language Model (LLM) is the standard architecture for building Speech LLMs. However, a structural misalignment exists between the encoder and the LLM.
arXiv AI
3d ago
Uni-Stream: A Unified Streaming Speech Language Model for Speech Synthesis and Recognition
Uni-Stream is a unified streaming speech language model that supports both speech recognition and synthesis within a single architecture. Built on compact pretrained speech models, it processes interleaved text and speech in real time and achieves performance comparable to task‑specific models through a two‑stage training approach. The authors also introduce a data annotation pipeline for streaming training and plan to release the generated transcriptions and word‑level timestamps, while analyzing how speech and text representations occupy distinct latent spaces in the model.
arXiv Computation and Language
Sep 15
Reducing the Output-Mode Gap in Speech Language Models via Joint-Output On-Policy Distillation
arXiv:2609.15313v1 Announce Type: cross Abstract: Autoregressive generation of interleaved text and acoustic tokens is a common approach to spoken-response generation in speech large language models....
arXiv AI
Jun 16
LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization
arXiv:2506. 16738v2 Announce Type: replace-cross Abstract: With the rapid progress of speech language models (SLMs), discrete speech tokens have emerged as a core interface between speech and text, enabling unified modeling across modalities.