← Back to all news
arXiv Computation and Language By Mateusz Modrzejewski, Tomasz Radzikowski

What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings

Read the original on arXiv Computation and Language →

The paper investigates what open audio‑language models actually attend to when judging generated music. By designing four probes—removing audio, swapping clips, changing clip order, and rewording instructions—the authors test whether the models respond to the music, the text prompt, or the question framing. Across five judges and thousands of clips, the results show that most models rely on the prompt alone, only a few follow the audio for prompt alignment, and instruction wording can shift correlations as much as changing the model itself.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Sep 15

The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speech

arXiv:2609.13150v1 Announce Type: cross Abstract: Reference-free quality predictors such as UTMOS, DNSMOS and SCOREQ are the de facto automatic evaluators for text-to-speech (TTS) and are increasingl...

By Antonis Asonitis, Juan Pablo Zuluaga Gomez, Francesco Verdini, Aref Farhadipour, Marzieh Razavi, Pierre-Edouard Honnet, Vijeta Avijeet
multimodal
More like this →
arXiv Computation and Language
Sep 21

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems

JudgeSense is a benchmark comprising 880 items from human‑labelled corpora, each presented under two differently worded instructions that ask the same question. The study evaluates 25 judges from six providers across four tasks, measuring how rewording affects agreement with the judge’s own verdicts. Results show that rewording reduces agreement on all tasks, with significant effects on two, and that stability varies across tasks and is not predicted by parameter count.

By Rohith Reddy Bellibatlu, Edward Raff, Wenbin Zhang
llmsagentsbenchmarks
More like this →
arXiv AI
Oct 16

Qwen-Audio-Jev: A Unified Interface for Audio Decisions

Qwen-Audio-Jev extends Qwen-Audio-3.1-ASR-Flash-Next with a dedicated decision head that unifies selection, binary judgment, and scoring into three primitives: CHOICE, NOUL, and SCORE. The model delivers competitive accuracies and error rates across 18 labeled configurations, achieving 78.70% on MMAU-mini, 84.06% on MMSU, 6.50% equal error rate on VoxCeleb1-O, and a 0.1062 normalized MAE on NISQA. Compared to its predecessor, Qwen-Audio-Jev offers roughly a 20‑fold inference speedup, enabling efficient high‑throughput audio decision‑making for tasks such as large‑scale screening, speaker verification, and automated quality assessment.

By Ying Fang, Zhifu Gao, Qing Gu, Jiajun He, Kangqi Jing, Xiangang Li, Yu Pu, Wupeng Wang
llmsefficiencymultimodalbenchmarks
More like this →
arXiv AI
Oct 19

SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferences

SpeechCritic learns a diagnostic speech judge from only about 300 human-labeled comparisons, calibrating a frontier audio-language model with non-binding hints that select acoustic measurements agreeing with human judgments. The approach improves dimension-level agreement by 6.3 points and reduces mismatch in Tie rates by 10.4 points, and training a 7B judge with SFT, OPD, and RL yields distinct behaviors, with RL producing more specific rationales. The method works across language pairs, demonstrating a path from limited human preferences to a diagnostic speech judge.

By Mingyue Huo, Shivam Mehta, Bhavin Jawade, Yinghong Lan, Haoqi Li
llmsreinforcement-learningfine-tuning
More like this →