← Back to all news
arXiv AI By Qianqi Yan, Yichen Guo, Ching-Chen Kuo, Shan Jiang, Hang Yin, Yang Zhao, Xin Eric Wang

OmniTrace: A Unified Framework for Generation-Time Source Attribution in Omni-Modal LLMs

Read the original on arXiv AI →

OmniTrace is a framework that attributes each sentence generated by an omni‑modal language model to specific source units—text chunks, images, or audio/video segments—by interpreting late‑layer attention as a mass distribution over these units, without requiring additional forward passes or per‑modality tuning. The method demonstrates that a single generation pass can recover 77–96 % of the effect of traditional leave‑one‑out attribution, which would otherwise need up to 41 extra passes. OmniTrace’s source selections are causally important for responses and align closely with human‑marked evidence, achieving higher AUROC scores than existing baselines on both document and video tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 18

AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models

AVTrace is a diagnostic suite designed to evaluate audio‑visual temporal reasoning in omni models. It covers tasks such as onset and span grounding, synchronization, next‑step prediction, cross‑modal localization, chain parsing, and event‑conditioned comprehension, providing 34,114 training examples and balanced development and test splits. Five open omni models were tested, all scoring below the majority‑label baseline on synchronization verification and showing low performance on chain parsing and event‑conditioned tasks, while parameter‑efficient temporal post‑training improved some metrics.

By Longyin Zhang, Parth Sakhare Mahendra, Chengwei Wei, Ning Zhang, Lim Ming Chong, Sirui He, Ai Ti Aw
multimodalbenchmarks
More like this →
arXiv Machine Learning
Sep 15

Omni-Streaming Thinking

arXiv:2609.15128v1 Announce Type: new Abstract: Streaming omni-modal models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support...

By Enjun Du, Siyi Liu, Ziyu Zheng, Jingyu Li, Yiwen Guo, Yongqi Zhang, Difan Zou
multimodalbenchmarkssafety
More like this →
arXiv AI
Jul 7

OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models

arXiv:2607. 03050v1 Announce Type: cross Abstract: Omni modal large language models (OmniLLMs) have attracted wide attention for their ability to jointly process audio and video, but they generate large token sequences under audio-visual inputs, leading to substantial inference cost.

By Shijie Cao, Qingyu Zhang, Boxi Yu, Yuzhong Zhang, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
llmsmultimodalbenchmarkssafety
More like this →
arXiv AI
Oct 1

LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception

LEAP is a framework for long audio‑video question answering that avoids encoding entire recordings by dividing them into fixed‑duration blocks. It performs a lightweight localization pass on each block to score short candidate windows, then pools the highest‑ranked windows for a single bounded answer pass, keeping the answer input and peak context independent of recording length. The method trains both a localization LoRA and an answer LoRA, supports causal streaming queries, and achieves significant performance gains over baseline models on multiple AVQA benchmarks.

By Juyi Lin, Zhiqiang Lao, Jiali Cui, Lin Zhao, Pu Zhao, Dichang Zhang, Arman Akbari, Yu Qi, Xinru Jiang, Yanzhi Wang, Heather Yu, Liang Peng
nlpfine-tuningmultimodalbenchmarks
More like this →