← Back to all news
arXiv Machine Learning By Mengzhe Geng, Max W. Y. Lam, Yuewen Cao, Shoukang Hu, Jing Xu, Marc Tessier, Xu Li

CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks

Read the original on arXiv Machine Learning →

CIViC (Conditional Input and Visual Interpretation Checks) is a four‑gate protocol designed to link interpretation to intervention, held‑out task comparison, or endpoint control in audio‑conditioned visual prediction. The authors evaluate CIViC on three proxy tasks—actor‑disjoint face‑raster tests, a synthetic audio‑feature mixture, and source‑video‑disjoint SpeakerVid—showing that different gates reveal distinct effects of audio manipulation on visual outputs. Across these tasks, CIViC demonstrates that output activation, held‑out error, and endpoint controls can support varied interpretations of audio influence on visual predictions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computer Vision
Sep 3

From Visual Cues to Spoken Narration: Rethinking Audio Description

The paper introduces Cue2Narrate, a two‑stage pipeline that jointly predicts what visual events to narrate and when to insert the narration in long, untrimmed movie clips. It uses a dual‑head audio‑visual localizer to identify visual cue and narration windows, followed by a LoRA‑adapted vision‑language model that generates concise audio descriptions, trained with a Description Ranking Loss. The authors also present the LongLSMDC benchmark, comprising up to 8‑minute clips, and show that Cue2Narrate outperforms video‑only and audio‑only baselines by 5–12 points in average mAP and improves AD generation over fine‑tuned base VLMs.

By Akshita Gupta, Aditya Arora, Federico Tombari, Marcus Rohrbach, Anna Rohrbach
fine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Sep 7

PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation

PRISM‑Bench is an audio‑centric diagnostic benchmark for text‑to‑audio‑video generation, built from 900 human‑verified samples. It evaluates audio along two axes—audio type (speech, music, sound) and sound‑source visibility (on‑screen vs. off‑screen)—across four perceptual dimensions (audio‑visual coherence, audio quality, audio expressiveness, and prompt following) using 35 fine‑grained criteria. The benchmark employs an enhanced MLLM‑as‑a‑Judge protocol that aligns strongly with human raters, revealing a performance gap between frontier and open‑source T2AV models and highlighting overfitting to perceptual fidelity while struggling with complex grounding and control tasks, especially for music and synchronized on‑screen audio.

By Yuchen Sun, Qian Yang, Jun Wang, Detai Xin, Guoqiao Yu, Guanglu Wan, Qi Jia
diffusionmultimodalbenchmarkssafety
More like this →
arXiv Computer Vision
Oct 8

TiTok: Audio-Visual LLM for Multi-Segment Temporal Grounding

TiTok is an audio‑visual large language model designed for multi‑segment temporal grounding in untrimmed videos. It introduces Time Token Interleaving (TTI) to align temporal perception with prediction, and uses decoupled reinforcement‑learning rewards (global, local, count, precision, format) optimized via GDPO. Evaluated on a new UnAV‑100 protocol with the CountF1 metric, TiTok achieves state‑of‑the‑art results (65.7 mIoU, 0.58 CountF1).

By Eunji Shin, Dahyun Choi, Seungyeon Jo, Yejin Hong, Jiyoung Lee
llmsreinforcement-learningmultimodalbenchmarks
More like this →