Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,954 stories · RSS feed

Hugging Face Trending Papers

Jul 21

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

Recent advancements in 3D Gaussian Splatting (3DGS) have enabled language-guided scene understanding. However, existing Referring 3D Gaussian Splatting (R3DGS) methods are fundamentally restricted to single-target queries.

llms computer-vision multimodal benchmarks

Hugging Face Trending Papers

Jul 21

Dual-Edged Homogeneous-Modality Similarity: Towards Visible-Infrared Modality-Incomplete Person Re-Identification with Modality Adaptive Matching

Visible-Infrared Person Re-Identification (VI-ReID) operates under a closed-world assumption, where queries and galleries are from heterogeneous modalities. However, in open-world scenarios, both sets are likely to contain homogeneous and heterogeneous modality images.

llms multimodal benchmarks

arXiv AI

Jul 21

Anticipate Before Acting: Future-State-Conditioned Vision-Language Navigation

arXiv:2607. 18042v1 Announce Type: cross Abstract: End-to-end vision-language navigation (VLN) with causal vision-language models can map instructions and egocentric observations directly to actions, but standard behavior cloning supervises only the next action and does not explicitly train the policy state to be predictive of future visual outcomes.

By Lingfeng Zhang, Zhanguang Zhang, Liheng Ma, Tongtong Cao, Yingxue Zhang

llms rag multimodal

arXiv AI

Jul 21

Reward-Driven LLM Agent Workflows: Synthesizing POMDP Routing and Self-Correction for Autonomous Decision-Making

arXiv:2607. 17038v1 Announce Type: new Abstract: This paper addresses key technical challenges in current large language model (LLM) agent applications, including long-horizon planning, sparse reward attribution, and dynamic environmental interaction, by designing and optimizing an intelligent agent workflow.

By Amez Amanj Ali, Kuo-Kun Tseng

llms agents reinforcement-learning robotics multimodal benchmarks safety

arXiv AI

Jul 21

PRISM: Multimodal Terrain Mapping for Rover Navigation in Unstructured Environments

arXiv:2607. 16366v1 Announce Type: cross Abstract: Robotic navigation in unstructured environments requires robust situational awareness to safely traverse hazards such as steep slopes and rocky terrain.

By Raul Castilla-Arquillo, Carlos Perez-del-Pulgar, Levin Gerdes, Alfonso Garcia-Cerezo, Miguel A. Olivares-Mendez

llms agents computer-vision robotics multimodal

arXiv Machine Learning

Jul 21

LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation

arXiv:2607. 17653v1 Announce Type: cross Abstract: Source-free universal domain adaptation (SF-UniDA) adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts, without access to source data.

By Jing Li, Pan Liu, Meng Zhao, Wanli Xue, Yanhong Yang, Xu Cheng, Fan Shi, Jianhua Zhang, Qinghua Hu, Shengyong Chen

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 21

Zoom In, Reason Out: Efficient Far-field Anomaly Detection in Expressway Surveillance Videos via Focused VLM Reasoning Guided by Bayesian Inference

arXiv:2604. 23724v3 Announce Type: replace-cross Abstract: Expressway video anomaly detection is essential for safety management.

By Xiaowei Mao, Bowen Sui, Weijie Zhang, Yawen Yang, Shengnan Guo, Shilong Zhao, Jiaqi Lin, Tingrui Wu, Youfang Lin, Huaiyu Wan

llms multimodal safety

arXiv Machine Learning

Jul 21

SGMCE: Segment-Grounded Morphological Concept Explanation for Malaria Parasite Species Identification in Thick Blood Smears

arXiv:2607. 16324v1 Announce Type: cross Abstract: Malaria diagnosis in endemic regions depends on species-level identification of Plasmodium parasites in thick blood smears, but deep learning detectors classify detections without providing morphological evidence for their predictions, limiting the ability of microscopists to audit those predictions at the case level.

By Ahmed Tahiru Issah, Charles B. Delahunt, Carine Mukamakuza

llms rag multimodal

arXiv AI

Jul 21

PhysAgent: Reflective Agentic Physics Control for Physically Plausible Video Generation

arXiv:2607. 16355v1 Announce Type: cross Abstract: Recent advances in physics-grounded video generation leverage physics simulation as a physical prior to guide video synthesis toward physically plausible outcomes.

By Qirui Li, Jinkun Hao, Yibo Li, Ran Yi, Paul L. Rosin, Yu-Kun Lai

llms agents diffusion multimodal safety

arXiv AI

Jul 21

What Do They See? Interpreting Complex Road Scenarios Through the Eyes of Vision-Language-Action Models for Safe and Trustworthy Autonomous Vehicle Learning

arXiv:2607. 16938v1 Announce Type: cross Abstract: End-to-end autonomous driving models are now able to navigate complex road scenarios, mapping raw sensor observations directly to observed paths for open-loop evaluation and often effective driving in closed-loop evaluation.

By Kalpana Panda, Wesley Maia, Vinti Agarwal, Ross Greer

agents robotics multimodal benchmarks safety

arXiv AI

Jul 21

Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

arXiv:2607. 18063v1 Announce Type: cross Abstract: LLM-based agents process external content, exposing them to prompt injection and multi-turn manipulation.

By Devina Jain, David Hartmann, Chuan Li

llms agents robotics multimodal benchmarks safety

arXiv AI

Jul 21

Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances

arXiv:2405. 12775v2 Announce Type: replace-cross Abstract: Discovering the semantics of multimodal utterances is essential for understanding human language and enhancing human-machine interactions.

By Hanlei Zhang, Hua Xu, Fei Long, Xin Wang, Kai Gao

multimodal benchmarks

arXiv AI

Jul 21

DocOCR-Eval: A Correction-Based Framework for OCR Tool Selection Without Ground Truth

arXiv:2607. 16203v1 Announce Type: cross Abstract: Document parsing is a foundational step for document understanding tasks such as visual question answering and key information extraction, as it transforms unstructured scanned images into structured representations by extracting textual, visual, and layout information.

By Zihan Xu, Puzhen Wu, Lawrence Chun Man Lau, Wei Liu, Sirui Li, Yifan Peng, Yihao Ding

llms computer-vision nlp multimodal benchmarks safety

arXiv AI

Jul 21

EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding

arXiv:2607. 17050v1 Announce Type: cross Abstract: GUI agents must reason about how actions transform interface states, but end-to-end success rates entangle this ability with perception, grounding, planning, and recovery.

By Yaohan Yang, Minglei Shi, Borui Zhang, Jie Zhou, Jiwen Lu

llms agents nlp multimodal benchmarks

arXiv AI

Jul 21

Teach it to stop, not just to click

arXiv:2607. 17136v1 Announce Type: cross Abstract: Agentic computer-use RL is reported in single runs, and those numbers mislead.

By Barada Sahu (Cabal AI), Shivesh Pandey (Para AI)

agents reinforcement-learning efficiency multimodal

arXiv AI

Jul 21

A Survey on Knowledge-Oriented Retrieval-Augmented Generation

arXiv:2503. 10677v3 Announce Type: replace-cross Abstract: Retrieval-Augmented Generation (RAG) has gained significant attention in recent years for its potential to enhance natural language understanding and generation by combining large-scale retrieval systems with generative models.

By Mingyue Cheng, Yucong Luo, Jie Ouyang, Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, Daoyu Wang, Enhong Chen

rag nlp multimodal benchmarks safety

arXiv Machine Learning

Jul 21

Self-Evolving Just-In-Time Memory for Proactive Embodied Safety

arXiv:2607. 16247v1 Announce Type: new Abstract: While Vision-Language Models (VLMs) have empowered embodied agents to execute complex household tasks, they struggle to proactively handle dynamically emerging hazards during closed-loop interactions.

By Bingrui Sima, Lizhong Wang, Xiaoya Lu, Kun He, Xiao Yang

llms agents robotics multimodal safety

arXiv AI

Jul 21

Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering

arXiv:2510. 01483v3 Announce Type: replace-cross Abstract: Vision-language models (VLMs) demonstrate strong image-level scene understanding, but reasoning over long egocentric video remains costly: because VLMs maintain no persistent memory or explicit spatial representation, all sampled frames must be re-processed for every new query.

By Mohamad Al Mdfaa, Svetlana Lukina, Timur Akhtyamov, Arthur Nigmatzyanov, Dmitrii Nalberskii, Sergey Zagoruyko, Gonzalo Ferrer

llms agents nlp robotics multimodal benchmarks

arXiv AI

Jul 21

ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding

arXiv:2607. 17884v1 Announce Type: new Abstract: Vision Language Models (VLMs) achieve strong reasoning with Chain-of-Thought (CoT) prompting but incur high sequential-generation cost, error accumulation, and limited self-correction.

By Keuntae Kim, Beomseok Lee, Hyunwoo Kim, Yong Suk Choi

llms diffusion multimodal benchmarks

arXiv AI

Jul 21

Do Speech Tokens Leak Voiceprints? Speaker Inversion Attacks Against End-to-End Speech Language Models

arXiv:2607. 16870v1 Announce Type: cross Abstract: End-to-end speech language models increasingly represent user speech with speech tokens rather than relying exclusively on cascaded ASR--LLM--TTS pipelines.

By Ye Lu, Yihan Yan, Zhaoyang Zhang, Zhitao Ou, Runze Liu, Li Liu, Shen Wang

llms rag multimodal

Get the daily briefing