arXiv Computation and Language September 17, 2026 By Pei-Jun Liao, Hung-Shin Lee, Wenze Ren, Kuo-Hsuan Hung, Hung-yi Lee, Hsin-Min Wang

Correlation-Guided Encoder Selection for Multi-Encoder Large Audio-Language Models

Read the original on arXiv Computation and Language →

The paper introduces CUES, a lightweight heuristic for selecting encoder combinations in large audio‑language models by estimating complementarity through Pearson correlations of single‑encoder performance profiles. Using a frozen SmolLM2‑135M backbone, CUES consistently identifies optimal encoder sets for each track on the XARES‑LLM benchmark without requiring fusion training or test data. On broad audio tasks, CUES selects a diverse trio of encoders, improving performance by 4.3% over Whisper‑medium, while on text generation it opts for a focused speech‑only pair, outperforming mHuBERT‑147 by 6.3%. The results illustrate how correlation signals guide a diversity–interference trade‑off across different task families.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Hugging Face Trending Papers

2d ago

Correlation-Guided Encoder Selection for Multi-Encoder Large Audio-Language Models

Multi-encoder fusion extends Large Audio-Language Models (LALMs) beyond speech-centric recognition, but selecting encoders via intuition or exhaustive search often introduces redundant representations...

llms fine-tuning multimodal benchmarks

arXiv AI

3d ago

Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations

arXiv:2609.14231v1 Announce Type: cross Abstract: Controllable synthesis of nonverbal vocalizations (NVVs) is es- sential for natural and expressive speech, but remains challeng- ing due to their aco...

By Ziyu Zhang, Yun Chen, Taihui Wang, Hanzhao Li, Qicong Xie, Rilin Chen, Zhixian Zhao, Lei Xie

fine-tuning

arXiv AI

Sep 10

Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs

arXiv:2609.05871v1 Announce Type: cross Abstract: Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-s...

By Song-ha Jo, Sehyun Lee, Soyoon Kim, Jaesik Choi, Sanghyuk Choi

llms multimodal safety

arXiv Computation and Language

1d ago

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

arXiv:2609.17981v1 Announce Type: cross Abstract: Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-R...

By Mohan Shi, Zilai Wang, Natarajan Balaji Shankar, Kaiyuan Zhang, Eray Eren, Abeer Alwan

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 1

BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations

arXiv:2606. 30700v1 Announce Type: cross Abstract: Self-supervised learning enables audio representations that transfer across domains and tasks.

By Ludovic K. Tuncay (IRIT-SAMoVA), Etienne Labb\'e (IRIT-SAMoVA), Thomas Pellegrini (IRIT-SAMoVA)

llms benchmarks

arXiv AI

6d ago

X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation

X-AuT is a progressive framework for compressing the audio encoder of speech large language models. It selects layer combinations via short behavioral probes and restores performance through representation alignment, cross‑scale distillation, scheduled student‑policy supervision, and LoRA finetuning, while keeping the language‑model backbone frozen. On ten Chinese–English benchmarks, reducing Qwen3‑ASR‑0.6B’s encoder from 18 to 16 layers lowers macro‑average error from 5.61% to 5.27%, and a 14‑layer model achieves 5.75% error with 20.7% fewer parameters.

By Haojun Zhang, Yi Zou, Min Chen, Qize Yu, Lianrui Fan, Xini Ding, Hao Li, Shuchang Zhou, Xianming Liu, Shiyu Huang

llms rag fine-tuning efficiency multimodal benchmarks safety

One email a morning, machine-written

Related stories

Correlation-Guided Encoder Selection for Multi-Encoder Large Audio-Language Models

Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations

Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations

X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation