arXiv Computation and Language

Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe

The paper introduces an Encoding Probe that reconstructs language model representations using interpretable features, addressing limitations of traditional decoding probes such as incomparable feature contributions and correlation effects. It evaluates this approach on text and speech transformer models, examining features from acoustics, phonetics, syntax, lexicon, and speaker identity. Findings reveal that speaker-related effects vary with training objectives and datasets, while syntactic and lexical features independently contribute to reconstruction, offering a complementary perspective on model interpretation.

arXiv AI

Sep 2

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

The paper investigates whether audio‑language models capture paralinguistic cues beyond spoken content. Using the Expresso dataset and four open‑source models, the authors trace how speaking style information is encoded in the late layers of the audio encoder but is degraded before reaching the final output. They find that some models are content‑driven while others are acoustic‑driven, revealing a gap between what is encoded and what is utilized in current audio‑language models.

By Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh, Bhiksha Raj

llms multimodal safety

arXiv AI

Aug 7

Bias Analysis of L2 Speaking Assessment Systems Using Concept Activation Vectors

arXiv:2608. 06300v1 Announce Type: new Abstract: Automatic speaking assessment systems are increasingly deployed in high-stakes settings to mark second language (L2) learners' speaking tests, making it critical to show that their scores depend on speaking proficiency rather than irrelevant speaker attributes such as first language (L1) or age.

By Arya Labroo, Mengjie Qian, Kate Knill

llms rag diffusion multimodal safety

arXiv Computation and Language

Sep 15

On the Interpretability of Whisper Encodings Using Sparse Autoencoders

arXiv:2605.12225v3 Announce Type: replace Abstract: While deep transformer-based models have advanced rapidly, their internal mechanisms remain largely a mystery. Recent work has prioritized understa...

By Dan Pluth, Zachary Nicholas Houghton, Yu Zhou, Vijay K. Gurbani

llms multimodal safety

arXiv Computation and Language

Sep 10

Do speech foundation models really learn words?

arXiv:2609.10434v1 Announce Type: new Abstract: Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the b...

By Robin Huo, Ewan Dunbar

llms multimodal

arXiv Computation and Language

Oct 1

VOSSA: Voiceprint Optimization for Streaming Speech Architectures

arXiv:2609.38887v1 Announce Type: cross Abstract: Real-time voice conversion (VC) systems commonly rely on pretrained speaker embeddings from automatic speaker verification (ASV) models. While effect...

By Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah, Ricardo Gutierrez-Osuna

rag

arXiv Computation and Language

3d ago

A Cross-lingual Comparison of Human and Classification Model Entrainment Behavior in Code-switched Speech Settings

arXiv:2607.25202v2 Announce Type: replace Abstract: Conversational entrainment is well-studied in monolingual and written contexts, but remains underexplored in spoken code-switching (CSW). We presen...

By Debasmita Bhattacharya, Siying Ding, Alayna Nguyen, Julia Hirschberg

llms agents benchmarks

arXiv Computation and Language

Oct 19

VocalGrad: Evaluating Acoustic Perception in Audio Language Models

arXiv:2610.23745v1 Announce Type: cross Abstract: Recent large audio language models (LALMs) have extended their capabilities beyond lexical speech understanding to tasks requiring paralinguistic and...

By Koki Ryu, Hitomi Yanaka

llms benchmarks

arXiv Computation and Language

Sep 15

Quantifying the Generation Modality Gap in Speech-Text Language Models

arXiv:2609.14743v1 Announce Type: new Abstract: Pure speech language models often lag behind text and speech-text language models in generating coherent content, but this gap is difficult to quantify...

By Ju-Chieh Chou, Jiawei Zhou, Karen Livescu

llms diffusion

arXiv AI

Jul 7

Probing Low-Level Acoustic Attribute Encoding in CLAP Audio Embeddings

arXiv:2607. 03806v1 Announce Type: cross Abstract: Audio foundation models are widely adopted as general-purpose feature extractors, yet the internal structure of their learned representations remains insufficiently understood.

By H\'ector Martel, Joe Hennessy-Priest, Taemin Cho

rag multimodal

arXiv Computation and Language

Oct 21

PhonoBridge: Error-Aware Phoneme-Mediated EMG-to-Text Reconstruction

arXiv:2610.27678v1 Announce Type: cross Abstract: Electromyography (EMG) provides a non-invasive means of decoding silently articulated speech into text, but accurate decoding remains challenging due...

By Wen Qiu, Jiacheng Zhang, Xuanda Chen, Siqi Cai, Haizhou Li

llms rag

arXiv AI

Jun 10

Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders

arXiv:2606. 10029v1 Announce Type: cross Abstract: Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tokens share a single residual stream.

By Nikita Koriagin, Georgii Aparin, Nikita Balagansky, Daniil Gavrilov

llms diffusion multimodal safety

arXiv AI

Jun 9

GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model

arXiv:2512. 20978v2 Announce Type: replace-cross Abstract: Language Model (LM)-based generative modeling has emerged as a promising direction for TSE, offering potential for improved generalization and high-fidelity speech.

By Haoyang Li, Xuyi Zhuang, Azmat Adnan, Ye Ni, Wei Rao, Shreyas Gopal, Eng Siong Chng, Boon Siew Han, Yuanjin Zheng

llms rag reinforcement-learning safety

Related stories