The paper introduces EASE, an encoder‑only model for Audio‑Visual Semantic Segmentation that eliminates redundant components found in prior Transformer‑based approaches. EASE achieves state‑of‑the‑art accuracy while running at up to 365 FPS—three times faster than previous models—and can be trained in under 11 GPU‑hours. The authors provide code, weights, and samples, positioning EASE as a scalable foundation for future research and real‑time applications.
arXiv:2608. 16285v1 Announce Type: cross Abstract: Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues.
By Zhaojin Fu, Yuyang Hong, Qi Yang, Zili Wang, Kun Ding, Shiming Xiang, Bin Fan
arXiv:2606. 02724v1 Announce Type: cross Abstract: Audio-visual speaker tracking aims to localize and track active speakers by leveraging auditory and visual cues, enabling fine-grained, human-centric scene understanding.
By Yaoting Wang, Yun Zhou, Zipei Zhang, Henghui Ding
arXiv:2607. 00687v1 Announce Type: cross Abstract: Comparing transformer backbones for image segmentation is confounded: each is paired with a different decoder, recipe, and pretraining, so reported differences rarely reflect the backbone itself.
By Tobias Christian Nauen, Anosh Billimoria, Federico Raue, Stanislav Frolov, Brian B. Moser, Andreas Dengel
arXiv:2511. 20973v2 Announce Type: replace-cross Abstract: Large Audio Language Models (LALMs) deliver strong performance across speech and audio tasks, but their audio encoders generate high-rate token sequences (e.
By Saurabhchand Bhati, Samuel Thomas, Hilde Kuehne, Rogerio Feris, James Glass
arXiv:2606. 27320v1 Announce Type: cross Abstract: Neural audio autoencoders have become a core component of compression, feature extraction, and generation.
By Dimitrios Bralios, Paris Smaragdis, Minje Kim
The paper introduces a new task called video object segmentation‑aware audio generation, which conditions sound synthesis on object‑level segmentation maps. It presents SAGANet, a multimodal generative model that uses visual segmentation masks, video, and textual cues to produce controllable audio for musical instruments, offering fine‑grained, visually localized control. The authors also release the Segmented Music Solos dataset of instrument performance videos with segmentation information to support this task and demonstrate that SAGANet outperforms current state‑of‑the‑art methods in controllable, high‑fidelity Foley synthesis.
By Ilpo Viertola, Vladimir Iashin, Esa Rahtu
SCAPES is a lightweight, resource‑efficient generative model that synthesizes high‑fidelity environmental sounds with high‑level semantic control. It operates on the continuous latent manifold of a neural audio codec, using a segmentation strategy and a Continuous Normalizing Flow to model latent trajectories. A 36‑million‑parameter instance can be trained on limited, uncurated data with a single consumer‑grade GPU, achieving convergence in roughly twice the source audio duration and enabling smooth semantic interpolation.
By Esteban Guti\'errez, Lonce Wyse, Frederic Font, Xavier Serra
arXiv:2609.10355v1 Announce Type: cross
Abstract: Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained lar...
By Killian Steunou, Yannis Tevissen, Moun\^im A. El Yacoubi
arXiv:2606. 30700v1 Announce Type: cross Abstract: Self-supervised learning enables audio representations that transfer across domains and tasks.
By Ludovic K. Tuncay (IRIT-SAMoVA), Etienne Labb\'e (IRIT-SAMoVA), Thomas Pellegrini (IRIT-SAMoVA)
The paper introduces an audio-first triage method for budgeted vision‑language captioning of untrimmed egocentric video. By selecting windows for a vision‑language model based on lightweight audio features before any video decoding, the approach reduces costly model calls. It achieves 4.0–10.8 percentage point improvements in action coverage across call rates, cuts 9–20% of VLM calls at matched coverage on EPIC‑KITCHENS‑100, and outperforms uniform sampling and recent visual keyframe selectors on Ego4D.
By Masoud Jalayer, Changyi Li, Yu Xiao
arXiv:2609.17509v1 Announce Type: cross
Abstract: Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing comput...
By Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe