Semantic-Critical Regions for Video-to-Audio Generation

The paper introduces SCR-V2A, a video-to-audio generation framework that focuses on Semantic-Critical Regions (SCRs)—localized visual cues directly related to sound events. SCR-V2A first identifies these regions via sound-related visual localization, then encodes them with an adapter-based dual visual encoder to produce a control representation that captures both global context and fine-grained sound evidence. This representation conditions an audio generation model, and experiments on in-domain and out-of-domain benchmarks show improved distribution matching and audio quality with few trainable parameters.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

• multimodal
• benchmarks
• safety