Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,631 stories · RSS feed

arXiv AI

Jun 2

Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Support

arXiv:2606. 00991v1 Announce Type: new Abstract: Transportation systems management and operations (TSMO) increasingly depends on timely interpretation of heterogeneous data, from various sensor streams, incident reports, traveler feedback, and visual observations.

By Siyan Li, Zehao Wang, Jiachen Li, Kanok Boriboonsomsin, Matthew J. Barth, Guoyuan Wu

llms efficiency multimodal benchmarks

arXiv AI

Jun 2

TIGER: Traceable Inference with Graph-Based Evidence Routing for Mitigating Hallucinations in Multimodal Generation

arXiv:2606. 00232v1 Announce Type: new Abstract: We study fact-level repair for multimodal generation, where a fluent output may contain specific facts that are not supported by the input.

By Kaixiang Zhao, Tianrun Yu, Shawn Huang, Porter Jenkins, Yushun Dong, Amanda Hughes

multimodal safety

arXiv Machine Learning

Jun 2

Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders

arXiv:2606. 00746v1 Announce Type: cross Abstract: Vision foundation models are bottlenecked by the quadratic cost of self-attention, which limits usable resolution and increases the cost of large-scale pretraining.

By Yitong Jiang, Hongjun Wang, Collin McCarthy, Hanrong Ye, David Wehr, Xinhao Li, Qi Dou, Tianfan Xue, Ka Chun Cheung, Simon See, Wonmin Byeon, Ke Chen, Kai Han, Jinwei Gu, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Sifei Liu

rag computer-vision efficiency multimodal

arXiv Machine Learning

Jun 2

HOIST: Humanoid Optimization with Imitation and Sample-efficient Tuning for Manipulating Suspended Loads

arXiv:2606. 00252v1 Announce Type: cross Abstract: Manipulating suspended payloads with humanoid robots is challenging because the robot can only influence an underactuated, oscillatory load through whole-body motion and intermittent contact.

By Songyang Liu, Shunyu Yao, Dingyuan Huang, Shuai Li

reinforcement-learning robotics multimodal

arXiv Machine Learning

Jun 2

IstGPT: LLM-based Anomaly Detection for Spatial-Temporal Graph in Industrial Systems

arXiv:2606. 01691v1 Announce Type: cross Abstract: Industrial Internet systems face increasing threats from sophisticated industrial control system (ICS) attacks, resulting in critical safety incidents.

By Yuchen Zhang, Ning Xi, Pengbin Feng, Shigang Liu, Jianfeng Ma, Yulong Shen, Yanan Sun, Xiaolin Zhou

llms robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Modeling Robotics Dataset Construction as an Artifact-Based Build Process

arXiv:2606. 00162v1 Announce Type: cross Abstract: Robotic systems generate large volumes of multimodal sensor data, but converting ROS bag recordings into machine learning datasets is often handled by ad hoc sequential scripts, creating engineering overhead and slow iteration cycles.

By Leon Pohl, Lukas Beer, George Sebastian, Mirko Maehlisch

robotics multimodal

arXiv Machine Learning

Jun 2

PaintBench: Deterministic Evaluation of Precise Visual Editing

arXiv:2606. 00188v1 Announce Type: cross Abstract: While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle.

By Kai Xu, Ellis Brown, Shrikar Madhu, Rob Fergus, He He, Saining Xie

robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 2

G2LoRA: Gradient Orthogonal Low-Rank Adaptation Framework for Graph Continual Learning on Text-Attributed Graphs

arXiv:2606. 01873v1 Announce Type: new Abstract: LLM-as-Aligner has emerged as a prevalent pre-training paradigm for Text-Attributed Graphs(TAGS), aligning graph and text modalities into a shared embedding space via CLIP-style contrastive learning.

By Yuhan Wang, Yibo Ding, Yutong Ye, Mufan Zhao, Wenbo Zhang, Ruijie Wang, Jianxin Li

llms rag fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging

arXiv:2606. 01717v1 Announce Type: new Abstract: Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization.

By Minsik Choi, Geewook Kim

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 2

AdaCodec: A Predictive Visual Code for Video MLLMs

arXiv:2606. 02569v1 Announce Type: cross Abstract: Video is temporally redundant: adjacent frames usually share most objects, background, and layout.

By Haowen Hou, Zhen Huang, Zheming Liang, Qingyi Si, Chenglin Li, Shuai Dong, Kele Shao, Ruilin Li, Dianyi Wang, Nan Duan, Jiaqi Wang

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

RDA: Reward Design Agent for Reinforcement Learning

arXiv:2606. 01672v1 Announce Type: new Abstract: Reinforcement learning has enabled the acquisition of impressive robotic skills, but typically requires hand-crafted reward functions that are slow to design and difficult to align with human intentions.

By Hojoon Lee, Ajay Subramanian, Ben Abbatematteo, Vijay Veerabadran, Pedro Matias, Karl Ridgeway, Nitin Kamra

llms agents reinforcement-learning robotics multimodal

arXiv AI

Jun 2

Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation

arXiv:2606. 00095v1 Announce Type: cross Abstract: Vision-Language Navigation (VLN) enables embodied agents to reach target locations in unseen environments by following language instructions.

By Kailing Li, Tianwen Qian, Lijin Yang, Yuqian Fu, Jingyu Gong, Xiaoling Wang, Liang He

llms agents robotics multimodal benchmarks

arXiv Machine Learning

Jun 2

Reason, Retrieve, Re-rank: A Zero-Shot Reasoning-Aware Framework for Composed Video Retrieval

arXiv:2606. 00910v1 Announce Type: cross Abstract: Composed Video Retrieval (CoVR) seeks the target video that results from applying a free-form textual modification to a reference video.

By Ali Alavi

llms multimodal

arXiv AI

Jun 2

ELF: A Family of Encoder-Free ECG-Language Models

arXiv:2601. 18798v2 Announce Type: replace-cross Abstract: ECG-Language Models (ELMs) extend recent advances in Multimodal Large Language Models (MLLMs) to automated ECG interpretation.

By William Han, Tony Chen, Chaojing Duan, Xiaoyu Song, Yihang Yao, Yuzhe Yang, Michael A. Rosenberg, Emerson Liu, Ding Zhao

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

ATLAS: Agentic Test-time Learning-to-Allocate Scaling

arXiv:2606. 01667v1 Announce Type: new Abstract: Test-time scaling has become a major way to improve large language model reasoning, but its orchestration has remained designer-engineered: a fixed sample budget, a fixed refinement loop, a fixed scoring rule, or a fixed search policy decides how compute is spent, leaving the model in charge of solving but not of orchestration.

By Peijia Qin, Qi Cao, Pengtao Xie

llms agents nlp multimodal benchmarks

arXiv AI

Jun 2

Deep Interest Mining for Intent-Enriched Semantic IDs in Multimodal Generative Recommendation

arXiv:2604. 20861v3 Announce Type: replace-cross Abstract: Semantic IDs (SIDs) provide the discrete item vocabulary used by generative recommendation, but their quality depends on what item evidence is preserved before quantization.

By Yangchen Zeng, Jinze Wang

llms rag reinforcement-learning efficiency multimodal

arXiv AI

Jun 2

MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction

arXiv:2604. 10169v2 Announce Type: replace Abstract: Trajectory prediction is a key component of autonomous driving systems because future motions directly affect collision checking, behavior planning, and control.

By Wenchang Duan, Zhenguo Gao, Jinguo Xian, Yi Shi

agents robotics efficiency multimodal safety

arXiv Machine Learning

Jun 2

Contrastive Representation Regularization for Vision-Language-Action Models

arXiv:2510. 01711v4 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models have shown strong capabilities in robot manipulation by leveraging rich representations from pre-trained Vision-Language Models (VLMs).

By Taeyoung Kim, Jimin Lee, Myungkyu Koo, Dongyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, Jinwoo Shin

llms robotics multimodal benchmarks

arXiv AI

Jun 2

Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs

arXiv:2606. 01215v1 Announce Type: cross Abstract: Current 3D spatial reasoning methods face a fundamental trade-off: neuro-symbolic 3D (NS3D) concept learners achieve interpretable reasoning through compositional programs but are constrained to closed-set concept vocabularies and simple programs; end-to-end 3D multi-modal LLMs (3D MLLMs) could handle complex natural language and open-vocabulary concepts but suffer from black-box reasoning without explicit spatial verification.

By Wentao Mo, Yang Liu

llms nlp reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Graph Transfer Learning via Shared Latent Geometry: Theory and Applications

arXiv:2606. 00716v1 Announce Type: new Abstract: Inference and control in engineered physical systems pay a heavy physics cost at deployment: state estimators, inverse-problem solvers, model-predictive controllers, schedulers, and observers are often not closed-form and must re-solve a numerical optimization per instance, with the operator re-supplied each time.

By Tong Wu, Andrew Campbell, Anna Scaglione

fine-tuning efficiency multimodal

Get the daily briefing