arXiv Machine Learning September 10, 2026 By Luis Vitor Zerkowski, Luiz Velho

BinauralVAE: Spatial Audio Reconstruction For World Models

Read the original on arXiv Machine Learning →

BinauralVAE is an open‑source pipeline that reconstructs spatial audio using various Variational Autoencoder architectures, including complex‑valued variants, to learn latent representations of binaural signals. The project builds on realistic acoustic data from a simulated robot navigating an environment, providing a foundation for audio‑centric world models. It aims to map the causal link between navigational actions and their acoustic outcomes, positioning sound as a complementary modality for spatial awareness.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

robotics

arXiv AI

Jun 10

Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding

arXiv:2606. 10738v1 Announce Type: cross Abstract: Recent multimodal large language models mainly process audio as monaural signals, thereby discarding the spatial cues contained in spatial audio for sound localization, spatial relation reasoning, and spatial scene understanding.

By Zhiyuan Zhu, Yixuan Chen, Yiwen Shao, Wenxiang Guo, Changhao Pan, Yu Zhang, Yuxiang Wang, Wei Liu, Houhua Zhang, Chengkuan Zeng, Wenbo Cheng, Yunxi Liu, Rui Yang, Steve Yves, Liefeng Bo, Zhou Zhao

llms nlp multimodal

arXiv AI

Jul 17

SceneBind: Binding What and Where Across Vision, Audio and Language

arXiv:2607. 15265v1 Announce Type: cross Abstract: We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language.

By Mingfei Chen, Zijun Cui, Ruoke Zhang, Hyeonggon Ryu, Eli Shlizerman

rag multimodal benchmarks safety

arXiv AI

Aug 18

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

arXiv:2608. 16285v1 Announce Type: cross Abstract: Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues.

By Zhaojin Fu, Yuyang Hong, Qi Yang, Zili Wang, Kun Ding, Shiming Xiang, Bin Fan

agents computer-vision robotics multimodal benchmarks safety

arXiv AI

2d ago

CTAN: Cycle-Temporal Attention Network for Embodied Audio-Visual Navigation

The paper introduces CTAN, a Cycle-Temporal Attention Network for audio‑visual embodied navigation. It proposes an Audio‑Visual Reconstruction Cross‑Attention module that uses bidirectional cycle‑consistency to strengthen spatial semantics across visual and acoustic modalities, and a Temporal Cross‑Modal Memory to fuse real‑time multimodal features with historical context. Experiments on Replica and Matterport3D show that CTAN outperforms prior methods in success rate, SPL, and scene navigation accuracy.

By Teng Liu, Yinfeng Yu

robotics multimodal benchmarks

arXiv AI

Aug 17

BAT: Learning to Reason about Spatial Sounds with Large Language Models

arXiv:2402. 01591v4 Announce Type: replace-cross Abstract: Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound.

By Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, David Harwath

llms

arXiv AI

Jul 29

Unlocking Spatial Grounding in Large Audio-Visual Retrieval models

arXiv:2607. 24786v1 Announce Type: cross Abstract: Weak supervision sets a practical regime for audio-visual sound source localization as dense spatial annotations are costly to obtain at scale.

By Hugo Malard, Michel Olvera, Sanjeel Parekh, Ga\"el Richard, Slim Essid, St\'ephane Lathuili\`ere

multimodal benchmarks safety

One email a morning, machine-written

Related stories

Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding

SceneBind: Binding What and Where Across Vision, Audio and Language

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

CTAN: Cycle-Temporal Attention Network for Embodied Audio-Visual Navigation

BAT: Learning to Reason about Spatial Sounds with Large Language Models

Unlocking Spatial Grounding in Large Audio-Visual Retrieval models