Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion
Read the original on arXiv Machine Learning →The paper introduces a novel four‑stream complex‑valued architecture for Open‑Vocabulary Audio‑Visual Event Localization (OV‑AVEL). By embedding visual, audio, and textual modalities into a shared space and computing complex‑valued similarities—real parts from standard embeddings and imaginary parts from companion streams (iHSV for vision and CycleGAN‑translated phase for audio)—the method learns fusion via a complex‑valued neural network while keeping encoders frozen. This approach achieves new state‑of‑the‑art results on OV‑AVEBench and a modified AVE dataset, outperforming previous fine‑tuned baselines by significant margins. "whyItMatters":"The proposed complex‑valued fusion strategy demonstrates that incorporating phase information and learning fusion in the complex domain can substantially improve open‑vocabulary event localization performance, setting new benchmarks for the task."
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.