arXiv AI
Aug 21
Towards Quantifying Benchmark Optimization in ASR Models
arXiv:2608. 19936v1 Announce Type: cross Abstract: Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities.
DecisionTraceBench: Tracking Decision-Critical Information in Multi-Party Spoken Conversations
Related stories
arXiv Computation and Language
Sep 25
Same Words, Different Actions: Paired Turn-Taking Evaluation under Rewritten Dialogue Contexts
The paper introduces ECHO, a paired diagnostic benchmark for evaluating Chinese real‑time spoken dialogue systems on turn‑taking. ECHO pairs examples that share the same overlap transcript but differ in preceding multi‑turn context, requiring either Yield or Keep actions, and also includes off‑talk cases to test unnecessary yielding. Experiments on four speech systems reveal that three systems over‑yield, correctly keeping the floor on fewer than 13% of backchannels, while the fourth system shows a more balanced performance, illustrating that interruption‑only evaluation can overestimate turn‑taking reliability.
Hugging Face Trending Papers
Sep 8
ConversationalVoice: Full-Duplex Speech Data from Real Conversations through Source-Faithful Reconstruction and Conversation-Grounded Expansion
Full-duplex speech models require training data that preserves turn-taking, overlap, interruption, and backchannel behavior, yet these signals are entangled across speakers in noisy real-world recordi...
arXiv Computation and Language
Oct 16
SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation
The paper introduces SITAR, a situational distortion metric for evaluating automatic speech recognition (ASR) systems. SITAR aggregates situational severity and cumulative extent into a unified score using a bounded recursive formulation, and is implemented as a lightweight BERT-family regressor that requires minimal runtime cost. The authors validate SITAR through extrinsic and intrinsic protocols, showing it tracks capability scaling and provides diagnostic utility across diverse datasets and models.
arXiv Computation and Language
Sep 22
MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing
arXiv:2609.23416v1 Announce Type: cross Abstract: Long-form audio performance is often summarized by context length and aggregate accuracy, obscuring how language, evidence, and task jointly shape di...
arXiv Computation and Language
Sep 25
Proactive for Uncertainty: Cause-Aware Error Diagnosis and Interactive Clarification for Spoken Dialogue Systems
The paper introduces a cause-aware error recovery framework for cascaded Automatic Speech Recognition – Large Language Model (ASR‑LLM) pipelines in Spoken Dialogue Systems. It replaces simple ASR confidence filtering with precision‑focused detectors that use deep ASR latent representations to classify token‑level errors into perception, comprehension, and deletion failures. This fine‑grained diagnosis enables the LLM to execute targeted, multi‑turn clarification strategies, leading to a more than two‑fold increase in recall on domain‑shift errors and significant reductions in word error rate and downstream task errors across varied accents, distortions, and domains.
arXiv Computation and Language
Sep 22
MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents
arXiv:2609.24812v1 Announce Type: new Abstract: Voice provides a natural and immediate interface for AI agents. Many settings in which voice agents could be useful, including meetings, households, an...
arXiv AI
Aug 28
When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue
The paper introduces ContraTalk, a benchmark that tests whether dialogue models truly use acoustic cues or rely on transcript shortcuts. It formalizes cross‑modal disagreement, creates conflict and consistent QA examples, and proposes an Audio Twin representation to expose acoustic evidence to models. Experiments show that while text‑only LLMs perform well on consistent cases, they falter on conflict cases, and AudioLLMs only partially mitigate this issue.
arXiv Computation and Language
Oct 1
When Does a Spoken Agent Have Enough Evidence to Act? The PACT-SLM Contract Test
arXiv:2609.38232v1 Announce Type: cross Abstract: Streaming spoken agents may take an external action before the available speech supports it, yet final-turn scores do not reveal whether each observe...
arXiv Computation and Language
Sep 11
SEAR: Segment-Evidence-Aware Routing for Weak-to-Strong Multilingual Speech MCQ
The paper presents SEAR, a system for Task 2 of the second Multilingual Conversational Speech Language Model Challenge. It adapts Qwen3-Omni-30B-A3B-Instruct with a segment‑evidence‑aware pipeline that converts timestamped ASR into event spans, expands and crops them, and then generates both semantic and acoustic multiple‑choice questions using Qwen3.6-27B and Gemini 3.1 Flash‑Lite. After rigorous checks, 359,825 verified MCQs across 21 languages and accents are produced, and the system achieves 90.92% accuracy on the official evaluation set.
arXiv Computation and Language
Sep 22
The Bairong System for MLC-SLM 2026: Dynamic Question-Aware Evidence Routing for Multilingual Conversational Speech Understanding
arXiv:2609.22214v1 Announce Type: new Abstract: Long multilingual conversational spoken question answering requires systems to balance long-range transcript semantics with sparse acoustic and speaker...
arXiv Computation and Language
Oct 22
Qwen-Audio-3.1-ASR Technical Report
The Qwen‑Audio‑3.1‑ASR family extends its predecessor with enhanced speech recognition and new speaker‑diarized transcription and general audio understanding features. It supports 30 languages and 16 Chinese dialects, offering instruction‑controlled capabilities such as historical‑context conditioning, hierarchical hotword customization, and native transcription polishing. The models deliver structured speaker‑diarized outputs, handle overlapping speech, and perform audio‑captioning, sound‑event detection, and audio question answering across multiple deployment variants.