Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,790 stories · RSS feed

arXiv AI

Aug 11

CircuitReason-1k: Benchmarking Long-Horizon Visual-to-Symbolic Reasoning inElectrical Circuits

arXiv:2608. 09374v1 Announce Type: new Abstract: Electrical circuit analysis requires more than recognizing components in an image.

By Xinqi Yang, Kang An, Tengyue Wang, Zhongyu Yang, Chenxu Du, Yuanchi Zhu, Hebao Zhu, Ziliang Wang, Faqiang Qian, Yunli Yang, Qibing Ren

llms multimodal benchmarks

arXiv AI

Aug 11

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence

arXiv:2608. 09281v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) perform strongly on engineering imagery, yet existing benchmarks mostly test drawing recognition, information extraction, or compliance checking, leaving open whether models can combine distributed visual evidence with engineering principles to reach a conclusion.

By Chenxu Du, Kang An, Tengyue Wang, Zhongyu Yang, Xinqi Yang, Yuanchi Zhu, Hebao Zhu, Ziliang Wang, Faqiang Qian, Yunli Yang, Qibing Ren

llms multimodal benchmarks safety

arXiv Machine Learning

Aug 11

Real Data Closes Synthetic-to-Real Gap in Optical Chemical Structure Recognition

arXiv:2608. 09100v1 Announce Type: new Abstract: Millions of chemical structures appear in patents and papers only as drawings, and using that information at scale requires reading the drawings.

By Yani Guan, Dengpan Dong, Zi Wei, Shuang Luo, Dan Hannah, Yumin Zhang, Kang Xu

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Aug 11

Auditing Medical Vision-Language Models on Chest Radiographs: Estimating Reference Agreement Across Institutions

arXiv:2608. 07550v1 Announce Type: cross Abstract: Vision-language models return structured chest-radiograph findings through interfaces exposing no confidence score, so a receiving institution cannot read off how far to trust an individual judgment.

By Pengyang Yu, Yiou Wang, Zhongping Dong, Sahraoui Dhelim, Chun-Mei Feng, M. Tahar Kechadi

llms multimodal

arXiv Machine Learning

Aug 11

Leveraging generative models to assist Monte Carlo sampling

arXiv:2608. 07648v1 Announce Type: cross Abstract: Sampling high-dimensional probability distributions is a central task in scientific computing, with applications ranging from Bayesian inference to statistical physics and molecular simulation.

By Marylou Gabri\'e

diffusion multimodal

arXiv Machine Learning

Aug 11

Auditing Instruction-Trajectory Mismatches in Multimodal Robot Demonstrations

arXiv:2608. 07895v1 Announce Type: cross Abstract: Robot demonstration datasets used to train vision-language-action policies can contain a subtle but harmful failure mode: trajectories that are behaviorally correct but paired with the wrong language instruction.

By Simon Holk, Ryosuke Takanami, Tatsuya Matsushima, Yusuke Iwasawa, Yutaka Matsuo, Yueh-Hua Wu, Kei Ota

robotics multimodal benchmarks

arXiv Machine Learning

Aug 11

Particle-Based Conformal Prediction for Contact-Aware Uncertainty Calibration in Stratified Configuration Spaces

arXiv:2608. 09166v1 Announce Type: cross Abstract: Reliable uncertainty representation is essential for deploying autonomous systems that interact with their environment, as robots must reason about how uncertainty arising from both stochasticity and model mismatch is impacted by contacts with obstacles (e.

By Lu\'is Marques, Kristian Popov, Dmitry Berenson

agents robotics multimodal

arXiv AI

Aug 11

EgoBrain: Synergizing Minds and Eyes For Human Action Understanding

arXiv:2506. 01353v3 Announce Type: replace Abstract: The integration of brain-computer interfaces (BCIs), in particular electroencephalography (EEG), with artificial intelligence (AI) has shown tremendous promise in decoding human cognition and behavior from neural signals.

By Nie Lin, Yansen Wang, Dongqi Han, Weibang Jiang, Jingyuan Li, Ryosuke Furuta, Yoichi Sato, Dongsheng Li

multimodal

arXiv AI

Aug 11

Decoy Images Amplify Caption-Mediated Defenses Against Encoded Jailbreaks

arXiv:2608. 01043v2 Announce Type: replace-cross Abstract: We report a counter-intuitive interaction between image inputs and existing black-box defenses on Vision--Language Models (VLMs): pairing an encoded jailbreak prompt with an unrelated decoy image can sharply lower attack success rate (ASR).

By Haoyu Zhang, Xiangchen Guan, Shibo Zheng, Mohammad Zandsalimy, Shanu Sushmita

llms multimodal safety

arXiv AI

Aug 11

CRUISE: Vision-Language Model-Guided Uncertainty-Aware Cross-Modal Sensor Fusion for Robust Autonomous Driving

arXiv:2608. 09202v1 Announce Type: new Abstract: Modern autonomous vehicles are equipped with multiple sensors, such as cameras, LiDAR, and radar, for comprehensive environmental perception.

By Junyao Wang, Yulin Xu, Yu Li, Pramod Khargonekar, Mohammad Abdullah Al Faruque

llms agents robotics multimodal

arXiv AI

Aug 11

Reading is not Reasoning: Bridging the Agentic Policy Gap in Vision-Text Compression

arXiv:2608. 08960v1 Announce Type: new Abstract: Multi-step language-model agents repeatedly process growing interaction histories, leading to substantial context costs.

By Cheng Fan, Junyi Zhou, Tingzhang Luo, RongJian Xu, Qiyanhui Lu, Mingjian Zhu, Hanting Chen, Jianyuan Guo

agents computer-vision reinforcement-learning efficiency multimodal

arXiv AI

Aug 11

COMEX: A Composition-Grounded Benchmark and Learning Framework for Explainable Aesthetic Image Cropping

arXiv:2608. 07570v1 Announce Type: cross Abstract: Explainable aesthetic image cropping requires not only localizing a visually pleasing crop but also explaining why it is preferred.

By Rui Yang, Wei Zhou, Dingyong Gou, Xiaohui Cui, Cong Li, Yinyin Gong, Yipo Huang, Jiliang Zhao

llms fine-tuning multimodal benchmarks

arXiv AI

Aug 11

Lingjing: A Simulation Testbed for Multi-Agent Embodied Tasks in Open-Ended Cities

arXiv:2608. 08045v1 Announce Type: new Abstract: Urban embodied intelligence requires coordination among heterogeneous agents (e.

By Xiaohe Li, Yiru Wang, Junhao Fan, Mingyuan Liu, Jie Huang, Kaixin Zhang, Jiahao Li, Chen Qian, Zide Fan

llms agents robotics multimodal

arXiv AI

Aug 11

Explore, Map, Remember, Decide: Are Embodied VLMs Ready for Safety-Critical Scenarios?

arXiv:2608. 08077v1 Announce Type: new Abstract: Theory of Space framework (ToS) assesses the spatial understanding of curiosity-driven Vision-Language Models (VLMs) under partial observability.

By Gabriele La Malfa, Nitay Alon, Emanuele La Malfa, Reuth Mirsky, Stefan Sarkadi

llms robotics multimodal safety

arXiv AI

Aug 11

Large Multimodal Agents for Intelligent Transportation Systems: Architectures, Evidence, and Deployment Challenges

arXiv:2608. 08184v1 Announce Type: new Abstract: Large multimodal agents (LMAs) are increasingly proposed for intelligent transportation systems (ITS), but existing studies often conflate multimodality, agency, empirical performance, and deployment readiness.

By Muhammad Ayub Sabir, Shaohong Zheng, Zhiyu Qu, Fatima Ashraf, Junbiao Pang

agents multimodal safety

arXiv AI

Aug 11

Discovering Diverse Planning Policies for Multimodal Embodied Agents with Quality-Diversity Optimization

arXiv:2608. 08523v1 Announce Type: new Abstract: Multimodal embodied agents are increasingly required to solve long-horizon tasks by integrating visual observations, textual goals, and interaction history into closed-loop decision making.

By Pengfei Xu, Yong Liu, Xiaoya Nan, Qiang Yang, Peilan Xu

agents robotics multimodal benchmarks

arXiv AI

Aug 11

Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models

arXiv:2608. 09227v1 Announce Type: new Abstract: Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive.

By Puneet Mathur, Manan Suri, Dinesh Manocha

llms nlp fine-tuning efficiency multimodal benchmarks

arXiv AI

Aug 11

CFD-Guided Detection of Concept Drift in Multimodal Physiologic Signals

arXiv:2608. 07759v1 Announce Type: cross Abstract: Cardiovascular AI models can classify clean elec- trocardiogram (ECG) signals, but real wearable signals change because of motion, breathing, posture, sensor contact, and true clinical deterioration.

By Farouk Ganiyu Adewumi, Timothy Oladunni, Rochak Ghimire, Kosisochukwu Ogbuanya, Sanaa Reeves, Sandy Akoy

multimodal

arXiv AI

Aug 11

Multimodal Federated Learning under Dual-Axis Modality Missingness

arXiv:2608. 09240v1 Announce Type: cross Abstract: Multimodal federated learning (FL) supports collaborative modeling in privacy-sensitive health-sensing and medical settings, but realistic deployments often exhibit dual-axis modality missingness: clients have different modality sets, and individual samples may contain only subsets of the modalities available locally.

By Adiba Orzikulova, Jaehyun Kwak, Jaemin Shin, Yunqi Guo, Xiaomin Ouyang, Guoliang Xing, Steven Euijong Whang, Sung-Ju Lee

multimodal safety

arXiv AI

Aug 11

Med-CRAFT: An Information System for Explainable and Configurable Construction of Multimodal Medical QA Datasets

arXiv:2512. 01045v2 Announce Type: replace Abstract: Data-intensive artificial intelligence applications increasingly rely on large-scale, high-quality, explainable, and reproducible datasets, yet the construction of such datasets often remains labor-intensive, weakly traceable, and difficult to configure.

By Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Bin Li

nlp multimodal

Get the daily briefing