Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,498 stories · RSS feed

arXiv AI

Jun 30

Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction

arXiv:2606. 29445v1 Announce Type: cross Abstract: Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks.

By Sunqi Fan, Qingle Liu, Runqi Yin, Meng-Hao Guo, Shuojin Yang

llms agents nlp multimodal benchmarks

arXiv AI

Jun 30

ManimAgent: Self-Evolving Multimodal Agents for Visual Education

arXiv:2606. 30296v1 Announce Type: new Abstract: Multi-round reflection lets agents built on large language models recover from failures within a single task, but each task remains an isolated episode: lessons learned across many reflection rounds on one task are discarded before the next begins.

By Wenjia Jiang, Zongyuan Cai, Yuanhang Shao, Chenru Wang, Boyan Han, Zhixue Song, Keyu Chen, Shengwei An, Xu Yang, Zhou Yang

llms rag agents multimodal

arXiv AI

Jun 30

Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models

arXiv:2603. 22876v2 Announce Type: replace-cross Abstract: Learning a generalist control policy for robotic manipulation typically relies on large-scale datasets.

By Ruixing Jin, Zicheng Zhu, Ruixiang Ouyang, Sheng Xu, Bo Yue, Zhizheng Wu, Guiliang Liu

reinforcement-learning robotics multimodal benchmarks

arXiv AI

Jun 30

EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generation

arXiv:2509. 12159v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models have demonstrated exceptional performance in UI2Code tasks, significantly enhancing website development efficiency.

By Jingyu Xiao, Zhongyi Zhang, Yuxuan Wan, Yintong Huo, Yang Liu, Michael R. Lyu

llms multimodal

arXiv AI

Jun 30

Demonstration-Free Robotic Control via LLM Agents

arXiv:2601. 20334v2 Announce Type: replace-cross Abstract: Robotic manipulation has increasingly adopted vision-language-action (VLA) models, which achieve strong performance but typically require task-specific demonstrations and fine-tuning, and often generalize poorly under domain shift.

By Brian Y. Tsui, Alan Y. Fang, Tiffany J. Hwu

llms agents robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 30

On the Faithfulness of Post-Hoc Concept Bottleneck Models

arXiv:2606. 30498v1 Announce Type: cross Abstract: Human decision-making interprets the world through high-level concepts, such as recognizing a bird by its belly color.

By Laines Schmalwasser, Jan Blunk, Niklas Penzel, Julia Niebling, Joachim Denzler

llms multimodal benchmarks

arXiv AI

Jun 30

Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models

arXiv:2511. 14900v2 Announce Type: replace-cross Abstract: Vision--language models (VLMs) have recently shown promise for assisting clinical reasoning in dermatological diagnosis.

By Zehao Liu, Weijieying Ren, Jipeng Zhang, Tianxiang Zhao, Jingxi Zhu, Xiaoting Li, Vasant G Honavar

llms reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 30

STEMGym: Benchmarking Sequential Decision-Making under Dose Budgets in Autonomous Electron Microscopy

arXiv:2606. 29592v1 Announce Type: new Abstract: A central premise of autonomous scientific imaging is that smarter navigation, whether Bayesian, RL-based, or otherwise adaptive, is the principal lever for sample-efficient acquisition.

By Can Polat, Erchin Serpedin, Mustafa Kurban, Hasan Kurban

llms agents reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 30

DataComp-VLM: Improved Open Datasets for Vision-Language Models

arXiv:2606. 28551v1 Announce Type: cross Abstract: Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies.

By Matteo Farina, Vishaal Udandarao, Thao Nguyen, Selim Kuzucu, Maximilian B\"other, Andreas Hochlehnert, Adhiraj Ghosh, Marianna Nezhurina, Karsten Roth, Joschka Struber, Yuhui Zhang, Sebastian Dziadzio, Elaine Sui, Soumya Jahagirdar, Dhruba Ghosh, Hasan Hammoud, Thomas De Min, Simone Caldarella, Jehanzeb Mirza, Sedrick Keh, Mehdi Cherti, Hilde Kuehne, Bernt Schiele, Serena Yeung-Levy, Muhammad Ferjad Naeem, Federico Tombari, Ana Klimovic, Elisa Ricci, Matthias Bethge, Sewoong Oh, Ameya Prabhu, Alessio Tonioni, Jenia Jitsev, Massimiliano Mancini, Ludwig Schmidt, Nikhil Parthasarathy

llms multimodal benchmarks

arXiv AI

Jun 30

CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training

arXiv:2603. 23559v2 Announce Type: replace-cross Abstract: GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices.

By Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 30

From Failure Taxonomy to Intervention: A Diagnostic Methodology for Industry-Scale AVLM in Video and Live-Streaming Platform Moderation

arXiv:2606. 30059v1 Announce Type: new Abstract: Industry-scale video and live-streaming moderation imposes requirements that are difficult to satisfy with generic pretrained public models or external APIs, including adaptation to platform-specific data distributions, policy-specific objectives, and product-level safety constraints.

By Shuchang Ye, Jinqiang Yu, Zhujun Xiao, Yajing Kong, Yist Y. Lin, Yang Ma, Jiaxi Liu, Xiaolei Xu, Zheng Yu

llms multimodal benchmarks safety

arXiv AI

Jun 30

CCRC: A Change-Aware Captioning and Reasoning Chain for Image Change Captioning and Segmentation

arXiv:2606. 28724v1 Announce Type: cross Abstract: Understanding and localizing subtle changes between paired images is critical for tasks such as surveillance and image editing.

By Jinhong Hu, Xiaoping Wang, Shuyin Huang, Guojin Zhong, Kaitai Liu, Kai Lu

llms computer-vision multimodal benchmarks

arXiv Machine Learning

Jun 30

BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

arXiv:2606. 30319v1 Announce Type: cross Abstract: Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience.

By Haitao Wu, Qirui Zhang, Zhouheng Yao, Shangquan Sun, Qihao Zheng, Mianxin Liu, Chi Zhang, Wanli Ouyang, Chunfeng Song, Changqing Zhang, Jiamin Wu

nlp multimodal benchmarks safety

arXiv Machine Learning

Jun 30

Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

arXiv:2604. 07753v2 Announce Type: replace-cross Abstract: Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts.

By Xiangyue Liu, Zijian Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Ping Tan

llms diffusion multimodal benchmarks

arXiv AI

Jun 30

GPTNT: Benchmarking Real-Time Collaboration Between Multimodal Agents on Keep Talking And Nobody Explodes

arXiv:2606. 28514v1 Announce Type: new Abstract: Multimodal models are increasingly deployed to solve tasks collaboratively with humans or other artificial agents.

By Amit Parekh, Sabrina McCallum, Kareem Al-Hasan, Malvina Nikandrou, Alessandro Suglia, Ioannis Konstas

agents multimodal benchmarks

arXiv Machine Learning

Jun 30

An Information-Geometric Justification for Composite Coherence in Event-Based Narrative Extraction

arXiv:2606. 29118v1 Announce Type: cross Abstract: Graph-based narrative extraction relies on a coherence function to score transitions between events, but the coherence metrics in current use are defined operationally and lack an information-theoretic foundation.

By Brian Keith-Norambuena

llms rag multimodal

arXiv AI

Jun 30

Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework

arXiv:2606. 29808v1 Announce Type: cross Abstract: Chart data extraction, which reverse-engineers data tables from chart images, is essential for reproducibility, analysis, retrieval, and redesign.

By Yuchen He, Peizhi Ying, Liqi Cheng, Kuilin Peng, Yuan Tian, Dazhen Deng, Yingcai Wu

llms multimodal benchmarks

arXiv Machine Learning

Jun 30

Road Maps as Free Geometric Priors: Weather-Invariant Drone Geo-Localization with GeoFuse

arXiv:2605. 14925v2 Announce Type: replace-cross Abstract: Drone-view geo-localization aims to match a query drone image, often captured under adverse weather conditions (e.

By Yunsong Fang, Tingyu Wang, Zhedong Zheng

robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 30

DreamForge-World 0.1 Preview: A Low-Compute Real-Time Controllable World Model

arXiv:2606. 30292v1 Announce Type: new Abstract: We present DreamForge-World 0.

By Daniyel Ayupov, Artur Markov-Tsoy

efficiency multimodal

arXiv Machine Learning

Jun 30

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

arXiv:2606. 28396v1 Announce Type: cross Abstract: Millimeter-wave (mmWave) radar perception is limited by data scarcity: models trained on existing radar datasets fail to generalize to new objects, environments, and sensing trajectories.

By Emily Bejerano, Federico Tondolo, Devang Gupta, Aaron Mano Cherian, Taeyoo Kim, Ayaan Qayyum, Xiaofan Yu, Xiaofan Jiang

llms robotics multimodal

Get the daily briefing