arXiv:2603. 09714v2 Announce Type: replace-cross Abstract: While multi-audio understanding is critical for large audio-language models (LALMs), it remains underexplored.
arXiv:2608. 11947v1 Announce Type: cross Abstract: Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge.
arXiv:2607. 13408v1 Announce Type: cross Abstract: Recent text-to-audio models generate high-quality audio, but often fail to follow instructions involving multiple sound events and temporal order.
The paper investigates position bias in large language models (LLMs) across a wide range of model sizes—from billions to trillions of parameters—and across five languages. It finds that while small-scale models and non‑English languages still exhibit noticeable position bias, larger models and the use of Chain‑of‑Thought prompting substantially reduce this bias. Additionally, the study shows that emerging thinking models with advanced reasoning capabilities are almost immune to position bias, indicating a promising but resource‑intensive path to mitigating this vulnerability.
arXiv:2609.35952v1 Announce Type: cross Abstract: We introduce HEAR (Human-recorded Evaluation of Audio-LLM bias by Real speakers), a large-scale, ecologically valid benchmark comprising 87k real hum...
arXiv:2606. 17417v1 Announce Type: cross Abstract: Large Audio Language Models (LALMs) achieve strong performance on a variety of audio understanding tasks but continue to struggle with temporal reasoning, a fundamental capability central to human auditory perception.
arXiv:2509. 22363v4 Announce Type: replace Abstract: Large Audio Language Models (LALMs) integrate audio encoders with pretrained Large Language Models to perform complex multimodal reasoning tasks.
arXiv:2610.35826v1 Announce Type: new Abstract: Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent e...
The paper investigates whether accuracy and order sensitivity in large language model (LLM) multiple‑choice question (MCQ) evaluation move together. Using proprietary and open‑weight models on MMLU and ARC‑Challenge, the authors measure order sensitivity with flip rate and recall standard deviation (RStd) and find that these metrics often diverge from accuracy and from each other. They conclude that accuracy alone is insufficient and recommend reporting both flip rate and RStd alongside accuracy.
arXiv:2609.14174v1 Announce Type: new Abstract: Asked to describe what one of six speakers in a recording talks about, audio language models describe the right one on 6 to 16% of trials, below the 16...
DEAF (Diagnostic Evaluation of Acoustic Faithfulness) is a benchmark comprising over 2,700 conflict stimuli that test Audio Multimodal Large Language Models (Audio MLLMs) across three acoustic dimensions: emotional prosody, background sounds, and speaker identity. The benchmark uses a controlled multi-level evaluation framework that gradually increases textual influence, from semantic conflicts to misleading prompts, to separate content-driven bias from prompt-induced sycophancy. Diagnostic metrics are introduced to measure how much models rely on textual cues versus acoustic signals, and an evaluation of seven Audio MLLMs shows that while models respond to acoustic variations, their predictions are largely driven by textual inputs, highlighting a gap between benchmark performance and true acoustic understanding.
AU-Harness is an open-source toolkit designed to streamline and standardize the evaluation of Audio Language Models (LALMs). It addresses three key limitations of existing frameworks: slow processing pipelines, lack of multi-turn dialogue support, and absence of a scalable, unified evaluation system. The toolkit achieves a 77% reduction in evaluation time, enabling large-scale studies and providing standardized protocols for fair model comparison and in-depth analysis of audio reasoning capabilities.