Hugging Face Trending Papers

Right Answers, Inconsistent Evidence: Source-Level Counterfactual Auditing of Large Audio-Language Models

arXiv AI
3d ago

Right Answers, Inconsistent Evidence: Source-Level Counterfactual Auditing of Large Audio-Language Models

The paper introduces TRACE, a method for auditing large audio‑language models (LALMs) by deleting the queried source and each rival sound in turn to test whether views containing the target sound consistently score higher than those without it. Across 8 LALMs and 3,456 controlled mixtures, 6.3% of correct answers fail this ordering, a failure that a simple target‑only score check misses 42.5% of the time. The failure rate rises from 4.9% to 8.4% as the target signal‑to‑noise ratio drops from +5 to –5 dB, and the issue persists even under loudness and re‑pairing controls.

By Yi Zhang, Yi Li, Hongwei Du
arXiv AI
Oct 12

Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination

The paper investigates how large audio‑language models (LALMs) hallucinate objects by affirming sounds that are not present. Using a black‑box causal test, the authors isolate and remove the claimed evidence region via spectral gating and compare the effect against a random‑window control across a 2×2×2 grid on three LALMs. The study finds that localization of evidence improves detection over random removal, that raw detection values are model‑dependent, and that contrastive decoding combining blank and language‑prior contrasts can significantly boost accuracy, but a unified causal‑grounding profile does not transfer between models.

By Mohammad Khalooei, Mohammad Sabokrou
Hugging Face Trending Papers
Oct 9

Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination

The paper investigates how large audio‑language models (LALMs) hallucinate objects by affirming sounds that are not present. Using a black‑box causal test, the authors isolate evidence regions via CLAP similarity, remove them with spectral gating, and compare against random windows across a 2×2×2 grid on three LALMs. The study finds that evidence localization improves detection over random removal, that raw detection scores are model‑dependent, and that a contrastive decoding strategy combining blank and language‑prior cues boosts Qwen2‑Audio accuracy from 57.1 % to 71.9 %, outperforming Audio‑Aware Decoding alone.

Hugging Face Trending Papers
Oct 27

Beyond Answer Changes: Matched Same-Class Controls for Local Audio Evidence

The paper investigates how audio-language models react to local sound‑event edits by comparing matched same‑class and different‑class replacements. Using consistent editing parameters across 600 synthetic soundscapes and three models, the study finds that different‑class replacements cause 49.7 to 60.9 percentage points more answer changes, while same‑class replacements keep accuracy within 5.2 percentage points of the original. Soundscape‑weighted selective accuracy for the inserted class ranges from 73.7 to 81.9%.

Hugging Face Trending Papers
Aug 3

Can Foundation Models Hear What Made That Sound? A Tiered Benchmark of Audio-Language Models and Traditional Classifiers for Closed-Set Sound Source Identification

We benchmark eleven audio classification methods: five task-aware closed-set LLMs (four Gemini models plus open-weight Kimi-Audio-7B-Instruct), four fixed-vocabulary taggers (YAMNet, PANNs, Whisper-AT, and SSLAM), a zero-shot audio-text model (CLAP), and an audio-grounded LLM (BAT). We evaluate them on a closed-set sound-source identification task over 2,242 clips spanning 23 fine-grained classes and 11 categories.

arXiv Computation and Language
Oct 22

Heard But Not Said: Prompting Understates Temporal Information of Musical Moments in Frozen Large Audio-Language Models

Large audio‑language models (LALMs) can understand music but struggle to pinpoint a described moment in time. Using a paired‑query probe on Slakh2100 stem activations, researchers attached state and change queries to the same audio span and varied preceding context. Across frozen LALMs, different extraction methods (full‑track prompting, windowed yes/no scoring, and attention) revealed that state queries perform well without context, while change queries benefit from preceding audio in most short‑segment models; however, 30‑second padded encoders show no comparable recovery, and prompting or attention alone expose almost none of this capability.

By Eunsik Shin
arXiv Computation and Language
Sep 16

EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models

EviScope is a new paired counterfactual benchmark that evaluates grounded language models by fixing the question while manipulating evidence—adding, removing, distracting, or contradicting it. The v1.1 dataset includes 40 four‑condition quartets with repaired counterfactual claims and span‑level support labels for automated assessment. Experiments on Qwen2.5‑7B, Llama 3.1 8B, and Gemini 3.5 Flash show that paired metrics reveal grounding behaviors hidden by simple answer accuracy, such as unsupported answers, conflict blindness, and incorrect non‑answer actions.

By Suryadeep Singh Deswal
arXiv AI
6d ago

Agreeing on the Answer but Not the Reason: Measurement Limits of LLM Reasoning Failure Taxonomies

The paper investigates how failure taxonomies for large language model (LLM) reasoning can be misleading, focusing on a category called Compositional Collapse. By measuring this category across twelve models and eleven languages on a multilingual math benchmark, the authors find that the reported fraction of failures is highly sensitive to the order of classification tests and that human annotators rarely agree with the automated inspector on this category. They also show that the inspector’s labeling is largely driven by the prompt rather than the model itself, and that the category is over‑represented compared to native speaker judgments.

By Madhu Sudhan Rao Beesetty