Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,396 stories · RSS feed

arXiv AI

Jul 7

Q-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answering

arXiv:2607. 03825v1 Announce Type: cross Abstract: Audio-Visual Question Answering (AVQA) extends classical VQA by requiring joint reasoning over video and synchronized audio.

By SungHun Kim, SeungJun Baek

nlp multimodal benchmarks

arXiv AI

Jul 7

MatPhaseBench: A Semantics-Guided Benchmark for Materials Phase Diagrams Understanding

arXiv:2607. 02934v1 Announce Type: cross Abstract: Materials phase diagrams are a core knowledge representation in materials science, encoding temperature,composition, phase stability, and phase transformation pathways, with their full understanding requiring thermodynamic mechanism analysis and scientific reasoning.

By Hanwen Wang, Sihan Liang, Zhiwei Liu, Yangang Wang, Wei Yan, Yuqin Liu, Zongguo Wang

multimodal benchmarks safety

arXiv AI

Jul 7

EGRA:Toward Enhanced Behavior Graphs and Representation Alignment for Multimodal Recommendation

arXiv:2508. 16170v2 Announce Type: replace-cross Abstract: MultiModal Recommendation (MMR) systems have emerged as a promising solution for improving recommendation quality by leveraging rich item-side modality information, prompting a surge of diverse methods.

By Xiaoxiong Zhang, Xin Zhou, Zhiwei Zeng, Yongjie Wang, Zhiqi Shen

llms efficiency multimodal safety

arXiv AI

Jul 7

Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

arXiv:2607. 03748v1 Announce Type: new Abstract: Unified multi-modal models (UMMs) have shown promising interleaved text-image reasoning capabilities, yet effectively optimizing such multi-turn generation via reinforcement learning (RL) remains an open challenge.

By Zican Hu, Xuyang Hu, Yiming Liu, Zuwei Long, Wei Liu, Yunzhuo Hao, Jiawei Gu, Linjie Li, Yu Cheng, Zhenhong Sun, Weibo Gu, Xing Sun, Zhi Wang

llms diffusion reinforcement-learning multimodal benchmarks

arXiv AI

Jul 7

Wan-Streamer v0.2: Higher Resolution, Same Latency

arXiv:2607. 04443v1 Announce Type: cross Abstract: We present Wan-Streamer v0.

By Lianghua Huang, Zhi-Fan Wu, Yupeng Shi, Wei Wang, Mengyang Feng, Junjie He, Chen-Wei Xie, Yu Liu, Jingren Zhou, Ang Wang, Bang Zhang, Baole Ai, Chen Liang, Cheng Yu, Chongyang Zhong, Jinwei Qi, Kai Zhu, Pandeng Li, Peng Zhang, Wenyuan Zhang, Xinhua Cheng, Yitong Huang, Yun Zheng, Yuxiang Bao, Yuzheng Wang, Zoubin Bi

llms agents multimodal

arXiv AI

Jul 7

BanglaMemeEvidence: A Multimodal Benchmark Dataset for Explanatory Evidence Detection in Bengali Memes

arXiv:2607. 03981v1 Announce Type: cross Abstract: Memes have become influential communication tools on social media, combining viral visuals with concise messaging to convey impactful ideas.

By Fatema Tuj Johora Faria, Mukaffi Bin Moin, Md. Mahfuzur Rahman, Pronay Debnath, Asif Iftekher Fahim, Faisal Muhammad Shah

computer-vision nlp multimodal benchmarks

arXiv AI

Jul 7

Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Models

arXiv:2607. 02575v1 Announce Type: cross Abstract: Vision-language models can perform new tasks without parameter updates through in-context learning (ICL), whose core mechanism is utilizing the support set for task induction.

By Kaiyun Yang, Ruilin Yang, Zhimin Yao, J. Wang, Wei Ge

llms multimodal benchmarks safety

arXiv AI

Jul 7

High-Fidelity One-Step Generative Visuomotor Policy via Recursive Correction, Frequency Consistency, and Contrastive Flow Matching

arXiv:2607. 03865v1 Announce Type: cross Abstract: Generative models such as diffusion and flow matching have advanced robotic visuomotor policies by modeling multimodal action distributions, but their multi-step sampling or ODE solving introduces inference latency.

By Yuran Chen, Xinye Cai, Zhonglin Gong, Yang Huang

diffusion robotics efficiency multimodal

arXiv AI

Jul 7

Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions

arXiv:2607. 03233v1 Announce Type: cross Abstract: The rapid growth of publicly available digital information has rendered manual open-source intelligence (OSINT) analysis insufficient for modern intelligence, cybersecurity, and cyber investigation.

By Eduardo Almeida Palmieri, Mohamed Chahine Ghanem, Dipo Dunsin, Zubair Baig, Ed de Quincey, Kim-Kwang Raymond Choo

llms rag agents nlp fine-tuning multimodal benchmarks safety

arXiv AI

Jul 7

PosterHarness: Turning Scientific Poster Generation into an Auditable Instruction-Following Benchmark

arXiv:2607. 03006v1 Announce Type: cross Abstract: Text-rich image models can now design poster-scale layouts, but we lack ways to measure whether they honor scientific communication contracts: legible labels, prescribed aspect ratios, and -- above all -- abstaining from fabricated scientific figures.

By Tianyi Yang, Dawei Fu, Youpeng Wu, Zixun Kou, Linrui Chen, Ruobing Jiang, Zijian Wang, Qiang Li

multimodal benchmarks

arXiv AI

Jul 7

Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models

arXiv:2607. 02819v1 Announce Type: cross Abstract: Cloud-edge Large Vision-Language Model (LVLM) inference enables efficient deployment by splitting computation between edge devices and cloud servers.

By Zikai Zhang, Rui Hu, Olivera Kotevska, Jiahao Xu

llms robotics multimodal benchmarks

arXiv AI

Jul 7

Attending to Multimodal Generation One Token at a Time

arXiv:2607. 03738v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context.

By Varun Gupta, Vineet Gandhi, Makarand Tapaswi

llms multimodal safety

arXiv AI

Jul 7

GeoSelect: Spatial-Program Execution for Training-Free Referring Remote Sensing Image Segmentation

arXiv:2607. 03869v1 Announce Type: cross Abstract: Referring remote sensing image segmentation isolates the object named by a natural-language expression in an aerial image.

By Yuhang Jiang, Guohui Deng, Miaozhong Xu, Chao Ruan, Jinling Zhao, Linsheng Huang

llms computer-vision multimodal

arXiv AI

Jul 7

Unified Audio Intelligence Without Regressing on Text Intelligence

arXiv:2607. 05196v1 Announce Type: cross Abstract: Audio intelligence involves understanding, reasoning about, and generating both audio and speech.

By Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

llms rag agents reinforcement-learning efficiency multimodal benchmarks safety

arXiv AI

Jul 7

MoP-JEPA: Hard-Assigned Predictor Mixtures for Stochastic JEPA World Models

arXiv:2607. 05238v1 Announce Type: new Abstract: JEPA world models predict the next latent state with a single deterministic predictor trained by latent regression.

By Zhi Song, Ximing Xing, Zhenchao Tang, hanbo Huang, Tianxu Lv, minghao Yang, Zhongzheng Niu, He Bing, Lusheng Wang, Jianhua Yao

rag multimodal

arXiv Machine Learning

Jul 7

Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions

arXiv:2604. 11730v4 Announce Type: replace-cross Abstract: Using behavioural science, health interventions focus on behaviour change by providing a framework to help patients acquire and maintain healthy habits that improve medical outcomes.

By Manuela Gonz\'alez-Gonz\'alez, Soufiane Belharbi, Muhammad Osama Zeeshan, Masoumeh Sharafi, Muhammad Haseeb Aslam, Lorenzo Sia, Nicolas Richet, Marco Pedersoli, Alessandro Lameiras Koerich, Simon L Bacon, Eric Granger

llms fine-tuning multimodal

arXiv Machine Learning

Jul 7

Agentic Very Long Video Understanding

arXiv:2601. 18157v3 Announce Type: replace-cross Abstract: The advent of always-on personal AI assistants, enabled by all-day wearable devices such as smart glasses, demands a new level of contextual understanding, one that goes beyond short, isolated events to encompass the continuous, longitudinal stream of egocentric video.

By Aniket Rege, Arka Sadhu, Yuliang Li, Kejie Li, Ramya Korlakai Vinayak, Yuning Chai, Yong Jae Lee, Hyo Jin Kim

llms rag agents multimodal benchmarks

arXiv Machine Learning

Jul 7

Pathways of Visual Information Flow in Vision-Language Models

arXiv:2607. 03358v1 Announce Type: cross Abstract: We study how visual information is routed in vision-language models (VLMs).

By Israfel Salazar, Stella Frank, Dan Oneata, Desmond Elliott, Constanza Fierro

llms multimodal benchmarks

arXiv Machine Learning

Jul 7

GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Text

arXiv:2312. 15320v3 Announce Type: replace-cross Abstract: Individuals with suspected rare genetic disorders often undergo multiple clinical evaluations, imaging studies, laboratory tests, and genetic tests over a prolonged period of time, a process commonly described as the diagnostic odyssey.

By Da Wu, Zhanliang Wang, Hongzhuo Chen, Jingye Yang, Cong Liu, Tzung-Chien Hsieh, Elaine Marchi, Justin Blair, Peter Krawitz, Chunhua Weng, Wendy Chung, Gholson J. Lyon, Ian D. Krantz, Jennifer M. Kalish, Kai Wang

llms multimodal benchmarks

arXiv Machine Learning

Jul 7

Is the Geometry Doing the Work? An Operating-Point Audit of Hierarchy in Hyperbolic Vision-Language Models

arXiv:2607. 05268v1 Announce Type: cross Abstract: Whether a hyperbolic representation model uses its geometry cannot be read off its curvature parameter: what matters is the dimensionless operating point $\sqrt{c}\rho$ and whether the radial and cone machinery is active there.

By Jaeyoung Kim, Eunseok Kim, Dongsuk Jang

llms multimodal

Get the daily briefing