Hugging Face Blog
Evaluating Audio Reasoning with Big Bench Audio
Read the original on Hugging Face Blog →The Flow has not summarised this story yet — read it at Hugging Face Blog.
Related stories
arXiv AI
Sep 28
ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval
ReasonAudio is a new benchmark designed to evaluate reasoning capabilities in text‑audio retrieval, addressing the gap left by existing semantic‑matching focused datasets. It tests four logical abilities—negation, temporal order, sound co‑occurrence, and sound duration—across five synthetic subtasks (1,000 queries over 10,000 composite clips) and one natural subtask (100 queries over 1,000 real‑world clips). Evaluation of 11 state‑of‑the‑art systems shows significant limitations, with the best model, OmniEmbed‑7B, scoring only 20.7 overall and 53.8% in a controlled setting, compared to 70.6% for its generative backbone and 95.6% for humans.
arXiv Computation and Language
5d ago
TORUS: A Unified Closed-Loop Benchmark for Unified Audio Models
The paper introduces TORUS, a closed‑loop benchmark designed to evaluate unified audio models on their ability to understand the audio they generate or edit. TORUS contains 1,053 questions across 21 tasks and 8 audio domain settings, covering all three audio types. The authors evaluate both unified models and cascades of specialized models, finding that the best unified models answer only about 52.4% of questions about their own outputs, while GPT‑Audio achieves 53.6%.
arXiv AI
Oct 7
Logbook: Extremely Long-form Audio Event Understanding
arXiv:2610.07338v1 Announce Type: cross Abstract: Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we intr...
arXiv AI
Sep 10
Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoning
arXiv:2510.11454v2 Announce Type: replace-cross Abstract: Recent advancements in large multimodal models (LMMs) have shown strong capabilities in audio understanding. However, most systems rely solel...
arXiv Computer Vision
Sep 1
TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos
arXiv:2605.07593v2 Announce Type: replace Abstract: Real-world audio-visual understanding requires chaining evidence that is sparse, temporally dispersed, and split across the visual and auditory str...
arXiv Machine Learning
Jun 18
Investigating Faithfulness in Large Audio Language Models
arXiv:2509. 22363v4 Announce Type: replace Abstract: Large Audio Language Models (LALMs) integrate audio encoders with pretrained Large Language Models to perform complex multimodal reasoning tasks.