← Back to all news
arXiv Computation and Language By Igor Sterner, Mirella Lapata, Alex Lascarides, Frank Keller

What, When, and How: Audio Description as Constrained Global Optimization

Read the original on arXiv Computation and Language →

The paper presents a new approach to automatic audio description (AD) that treats the task as a constrained global optimization problem. It jointly decides what visual content is narratively important, when it can be spoken without overlapping dialogue, and how to phrase it within time limits. Using large language models for salience estimation and a mixed‑integer linear program for scheduling, the system outperforms prior methods on the REFRAMED benchmark, especially in temporal placement and narrative relevance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computer Vision
Sep 3

From Visual Cues to Spoken Narration: Rethinking Audio Description

The paper introduces Cue2Narrate, a two‑stage pipeline that jointly predicts what visual events to narrate and when to insert the narration in long, untrimmed movie clips. It uses a dual‑head audio‑visual localizer to identify visual cue and narration windows, followed by a LoRA‑adapted vision‑language model that generates concise audio descriptions, trained with a Description Ranking Loss. The authors also present the LongLSMDC benchmark, comprising up to 8‑minute clips, and show that Cue2Narrate outperforms video‑only and audio‑only baselines by 5–12 points in average mAP and improves AD generation over fine‑tuned base VLMs.

By Akshita Gupta, Aditya Arora, Federico Tombari, Marcus Rohrbach, Anna Rohrbach
fine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Oct 31

Mind The Gap (MTG-AD): Dataset and Fine-grained Evaluation for End-to-End Inline Audio Descriptions

The paper introduces MTG-AD, a 36‑hour dataset of professionally annotated inline audio descriptions for videos, created by blind and sighted professionals. It also presents MTG‑AD‑Eval, a reference‑free evaluation framework that measures how well generated descriptions fit within a video’s natural silences, providing fine‑grained metrics aligned with expert guidelines. Experiments show that while large multimodal language models can reduce speech overlaps, gaps remain in pacing, and users find the model‑generated descriptions highly beneficial for comprehension.

By Subhashini Venugopalan, Jenny Tan, Taylor Roper, Jimmy Tobin, Anton Kast, Alicia Martin, Sam Sepah, Amy Pavel
llmsmultimodalbenchmarks
More like this →