arXiv AI
SMART: Semantic Segment Mapping and Assessment of Recognition Transcripts with Contextual Rubrics
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 21
Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER
arXiv:2609.21663v1 Announce Type: new Abstract: Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equal...
arXiv Computation and Language
Oct 16
SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation
The paper introduces SITAR, a situational distortion metric for evaluating automatic speech recognition (ASR) systems. SITAR aggregates situational severity and cumulative extent into a unified score using a bounded recursive formulation, and is implemented as a lightweight BERT-family regressor that requires minimal runtime cost. The authors validate SITAR through extrinsic and intrinsic protocols, showing it tracks capability scaling and provides diagnostic utility across diverse datasets and models.
arXiv AI
Jun 3
Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling
arXiv:2606. 02837v1 Announce Type: cross Abstract: Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited.
arXiv AI
Sep 1
No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus
The study examined whether providing full prompt-level context to a large multimodal model would improve speech transcription accuracy on a production oral‑history corpus. Using a preregistered within‑item paired ablation, the authors found that adding context did not produce a detectable change in side‑level word error rate (WER) for either gpt‑4o‑transcribe or gemini‑2.5‑flash. The results suggest that context alone may not be sufficient to enhance aggregate transcription accuracy, and that finer‑grained, sequence‑aligned metrics are needed to evaluate such mechanisms.
arXiv Machine Learning
Sep 21
Beyond WER: Entity and Disfluency Recall in Accented Conversational ASR
The paper introduces a three‑stage pipeline to improve accented conversational ASR for speakers from India, Indonesia, and Latin America. It uses heuristic SQL filters to curate entity‑rich training data, regional LoRA adapters fine‑tuned on Qwen2.5‑Omni‑3B to generate both verbatim and corrected transcripts, and a six‑category error taxonomy validated by an LLM judge. The approach raises entity recall to 80‑85% and filler recall to 76‑86%, while keeping WER low (6‑10%) and outperforming Whisper and a commercial ASR on entity recall.
arXiv Computation and Language
Aug 27
Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence
The study investigates how prior scores influence large language model (LLM) judgments in the LLM-as-a-Judge paradigm. By testing three prompt conditions—no metadata, revision framing, and anchored metadata containing prior scores—the authors find that prior scores systematically bias evaluations, shifting ratings toward those scores across 192,000 attempts. The bias also affects categorical decisions, blocking 48% of error corrections and flipping 10.18% of correct judgments, and is not mitigated by Chain-of-Thought or a warning, underscoring the need for careful context engineering.