← Back to all news
arXiv AI By Xiaofei Sun, Peng Wang, Huayao Liu, Xiaoxiao Xu, Jun Song, Deqing Li, Lin Qu

When Does Sharing Work? Scaling a Shared SigLIP2 Across Image, Video, Audio, and Audio-Visual Representations

Read the original on arXiv AI →

The paper introduces Omni-Encoder, a single SigLIP-2 Vision Transformer that jointly processes image, video, audio, and audio‑visual data using fully shared parameters. It replaces separate modality encoders and anchor‑based alignment with a unified architecture that tokenizes audio, applies multimodal rotary position embeddings, and trains all modalities end‑to‑end through alternating‑modality batches. Omni‑Encoder achieves competitive performance on 16 benchmarks across all four modalities, outperforming individually trained counterparts, and the authors plan to release the code and models publicly.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 16

MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

The paper introduces Modality Self-Play (MSP), a training framework that enables multimodal large language models to learn cross‑modal interactions without requiring paired non‑text data. MSP uses proxy‑mediated textual descriptions of unseen modalities as semantic bridges, allowing a model to generate and align these proxies with observed modalities. Experiments show that MSP achieves state‑of‑the‑art or competitive performance on several multimodal benchmarks and demonstrates effective zero‑shot modality composition through learned bridges and anchor‑supporting proxy descriptions.

By Robert Moseley, Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, Salman Asif
llmsreinforcement-learningmultimodalbenchmarkssafety
More like this →
arXiv Machine Learning
Jul 14

Empowering Long-form Omni-modal Understanding with Robust Audio Perception

arXiv:2607. 10299v1 Announce Type: new Abstract: Recent advances in large-scale multimodal models have drivenremarkable progress in vision-language tasks; however, comprehensiveomni-modal understanding remains under-explored, largely due to thescarcity of datasets with rich, explicitly aligned auditory cues.

By Kaiying Yan, Luoyi Sun, Xiao Zhou, Weidi Xie
fine-tuningmultimodalbenchmarks
More like this →