arXiv AI

Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions

The paper introduces Hybrid Search, a method that refines warm-initialized large language model (LLM) based automatic speech recognition (ASR) systems by exploiting interactions between ASR hidden states and the base LLM’s hidden states. By identifying tokens with high semantic dependence and selectively correcting them, the approach surpasses traditional global LLM‑correction techniques such as rescoring and late fusion. The study demonstrates that even after warm initialization, LLM‑based ASR models can further benefit from their base LLM during inference.

arXiv Computation and Language

Sep 15

Merging the Knowledge of LLMs for Automatic Speech Recognition

arXiv:2609.15743v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems, trained on paired speech-text data, have been improved by leveraging language models (LMs) trained on text-...

By Hayato Futami, Tatsuya Kawahara

llms fine-tuning efficiency multimodal

arXiv AI

Jun 16

LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization

arXiv:2506. 16738v2 Announce Type: replace-cross Abstract: With the rapid progress of speech language models (SLMs), discrete speech tokens have emerged as a core interface between speech and text, enabling unified modeling across modalities.

By Daejin Jo, Jeeyoung Yun, Byungseok Roh, Sungwoong Kim

llms nlp efficiency multimodal safety

arXiv Computation and Language

Sep 17

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

arXiv:2609.17981v1 Announce Type: cross Abstract: Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-R...

By Mohan Shi, Zilai Wang, Natarajan Balaji Shankar, Kaiyuan Zhang, Eray Eren, Abeer Alwan

llms fine-tuning multimodal benchmarks

arXiv AI

Aug 21

Towards Quantifying Benchmark Optimization in ASR Models

arXiv:2608. 19936v1 Announce Type: cross Abstract: Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities.

By Theo Lebryk, David Ayllon, Alice Baird, Jakub Piotr C{\l}apa, Jens Madsen, Panagiotis Tzirakis

multimodal benchmarks

arXiv Computation and Language

3d ago

Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors

arXiv:2610.24809v2 Announce Type: replace Abstract: Speech recognition in real time in far-field conversational settings still incurs significant errors. Identifying potential error spans during stre...

By Sankalp Parashar, Preethi Jyothi, Sunita Sarawagi

llms multimodal

arXiv Computation and Language

Oct 7

HINTT Submission to the 2nd MLC-SLM Challenge: Comparing Cascaded and Unified Approaches to Diarization and ASR

arXiv:2610.08063v1 Announce Type: cross Abstract: This paper presents the HINTT system submitted to the 2nd Challenge and Workshop on Multilingual Conversational Speech Language Model (MLC-SLM). We a...

By Takanori Ashihara, Kohei Matsuura, Masato Mimura

llms fine-tuning multimodal

arXiv Computation and Language

Sep 1

A Simple Method to Enhance Pre-trained Language Models with Speech Tokens for Classification

arXiv:2512.07571v3 Announce Type: replace Abstract: This paper presents a simple method that allows to easily enhance textual pre-trained large language models with speech information, when fine-tune...

By Nicolas Calbucura, Jose Guillen, Valentin Barriere

llms rag nlp fine-tuning multimodal

Hugging Face Trending Papers

Sep 16

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Sp...

llms fine-tuning multimodal benchmarks

arXiv AI

3d ago

Abstract Compression of Conversational Audio Context for LLM-based ASR

arXiv:2603.26246v2 Announce Type: replace-cross Abstract: Standard LLM-based speech recognition systems process utterances in isolation, limiting their ability to leverage conversational context. Her...

By Shashi Kumar, Esa\'u Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Ba\~neras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke

llms rag multimodal

arXiv Computation and Language

Sep 22

Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR

arXiv:2604.06487v2 Announce Type: replace Abstract: Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR arch...

By Thibault Ba\~neras-Roux, Sergio Burdisso, Esa\'u Villatoro-Tello, Dairazalia S\'anchez-Cort\'es, Shiran Liu, Severin Baroudi, Shashi Kumar, Hasindri Watawana, Manjunath K E, Kadri Hacioglu, Petr Motlicek, Andreas Stolcke

llms fine-tuning multimodal safety

arXiv AI

Aug 11

VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference

arXiv:2608. 08569v1 Announce Type: new Abstract: Recent advancements in Speech Large Language Models have demonstrated remarkable capabilities in understanding complex audio tasks.

By Wenxu Jia, Dongjie Fu, Xize Cheng, Fangming Feng, Linjun Li, Wenshi Chen, Yingming Li, Zhou Zhao, Tao Jin

llms efficiency multimodal benchmarks safety

arXiv AI

Aug 10

MetaSICL: Globalizing Auditory LLMs for Underserved Speakers and Languages via Meta Speech In-Context Learning

arXiv:2601. 18904v3 Announce Type: replace-cross Abstract: Generative AI for speech and audio is increasingly expected to serve users across languages, cultures, and communities, yet current auditory Large Language Models (LLMs) are still largely trained and evaluated on high-resource data.

By Haolong Zheng, Siyin Wang, Zengrui Jin, Mark Hasegawa-Johnson

llms reinforcement-learning fine-tuning multimodal

Related stories