arXiv Computation and Language

Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe

The paper introduces an Encoding Probe that reconstructs language model representations using interpretable features, addressing limitations of traditional decoding probes such as incomparable feature contributions and correlation effects. It evaluates this approach on text and speech transformer models, examining features from acoustics, phonetics, syntax, lexicon, and speaker identity. Findings reveal that speaker-related effects vary with training objectives and datasets, while syntactic and lexical features independently contribute to reconstruction, offering a complementary perspective on model interpretation.

arXiv AI

Sep 2

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

The paper investigates whether audio‑language models capture paralinguistic cues beyond spoken content. Using the Expresso dataset and four open‑source models, the authors trace how speaking style information is encoded in the late layers of the audio encoder but is degraded before reaching the final output. They find that some models are content‑driven while others are acoustic‑driven, revealing a gap between what is encoded and what is utilized in current audio‑language models.

By Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh, Bhiksha Raj

llms multimodal safety

arXiv AI

Aug 7

Bias Analysis of L2 Speaking Assessment Systems Using Concept Activation Vectors

arXiv:2608. 06300v1 Announce Type: new Abstract: Automatic speaking assessment systems are increasingly deployed in high-stakes settings to mark second language (L2) learners' speaking tests, making it critical to show that their scores depend on speaking proficiency rather than irrelevant speaker attributes such as first language (L1) or age.

By Arya Labroo, Mengjie Qian, Kate Knill

llms rag diffusion multimodal safety

arXiv Computation and Language

3d ago

Do speech foundation models really learn words?

arXiv:2609.10434v1 Announce Type: new Abstract: Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the b...

By Robin Huo, Ewan Dunbar

llms multimodal

arXiv AI

Jul 7

Probing Low-Level Acoustic Attribute Encoding in CLAP Audio Embeddings

arXiv:2607. 03806v1 Announce Type: cross Abstract: Audio foundation models are widely adopted as general-purpose feature extractors, yet the internal structure of their learned representations remains insufficiently understood.

By H\'ector Martel, Joe Hennessy-Priest, Taemin Cho

rag multimodal

arXiv AI

Jun 10

Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders

arXiv:2606. 10029v1 Announce Type: cross Abstract: Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tokens share a single residual stream.

By Nikita Koriagin, Georgii Aparin, Nikita Balagansky, Daniil Gavrilov

llms diffusion multimodal safety

arXiv AI

Jun 9

GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model

arXiv:2512. 20978v2 Announce Type: replace-cross Abstract: Language Model (LM)-based generative modeling has emerged as a promising direction for TSE, offering potential for improved generalization and high-fidelity speech.

By Haoyang Li, Xuyi Zhuang, Azmat Adnan, Ye Ni, Wei Rao, Shreyas Gopal, Eng Siong Chng, Boon Siew Han, Yuanjin Zheng

llms rag reinforcement-learning safety

arXiv AI

Jun 29

From Signals to Transfer: A Factorised Study of Probe-Based Uncertainty Estimation in Large Language Models

arXiv:2606. 27679v1 Announce Type: cross Abstract: Probe-based uncertainty estimation (UE) has emerged as a prominent approach to detect hallucinations in Large Language Models (LLMs) by learning uncertainty from internal model signals.

By Ponhvoan Srey, Xiaobao Wu, Cong-Duy Nguyen, Quang Minh Nguyen, Duc Anh Vu, Anh Tuan Luu

llms benchmarks safety

arXiv AI

Sep 1

A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretability

arXiv:2608.29034v1 Announce Type: cross Abstract: A wide range of methods have been proposed for interpreting language models, delivering important insights into their inner workings. However, differ...

By Zhang Enyan, R. Thomas McCoy

llms safety

arXiv Machine Learning

3d ago

Interleaved Speech Language Models Latently Work In Text

arXiv:2606.22473v2 Announce Type: replace-cross Abstract: Speech language models (SLMs) increasingly combine speech and text, often by interleaving their tokens within a single sequence. Yet how thes...

By Talia Sternberg, Gallil Maimon, Yossi Adi

llms diffusion multimodal

arXiv AI

1d ago

Flexible and Interpretable Accent Distance Measurements

The paper introduces a method for measuring accent differences that balances interpretability and practicality. It proposes using articulatory representations obtained via articulatory inversion as an interpretable basis for accent comparison, while employing optimal transport to compare accents across any type of recording. This approach aims to overcome the limitations of traditional phonetic analyses and embedding‑based methods, which are either time‑consuming or non‑interpretable.

By Charles McGhee, Mark J. F. Gales, Kate M. Knill

rag multimodal

arXiv AI

Jun 17

L-Proto: Language-Aware Episodic Prototypical Training for Multilingual Speaker Verification

arXiv:2606. 17416v1 Announce Type: cross Abstract: Multilingual speaker verification remains challenging because language-dependent acoustic variability causes speaker identity to become entangled with linguistic characteristics, degrading generalization across languages.

By Hyung-Seok Oh, Deok-Hyeon Cho, Seung-Bin Kim, Seong-Whan Lee

rag fine-tuning benchmarks

arXiv AI

Jun 16

LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization

arXiv:2506. 16738v2 Announce Type: replace-cross Abstract: With the rapid progress of speech language models (SLMs), discrete speech tokens have emerged as a core interface between speech and text, enabling unified modeling across modalities.

By Daejin Jo, Jeeyoung Yun, Byungseok Roh, Sungwoong Kim

llms nlp efficiency multimodal safety

Related stories