Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,631 stories · RSS feed

arXiv Machine Learning

Jun 3

FinStressTS: A Parametric Synthetic Benchmark for Time-Series Forecasting in Finance

arXiv:2606. 03184v1 Announce Type: cross Abstract: Financial forecasting is difficult due to low signal-to-noise ratios, latent factors, heavy tails, regime shifts, and jumps.

By Jiaze Sun, Kelvin J. L. Koa, Ruiyang Ni, Yize Liu, Haonan Chen, Ke-Wei Huang

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 3

A Fast Screening Approach for High-dimensional Outcomes and High-dimensional Predictors

arXiv:2606. 03018v1 Announce Type: cross Abstract: Modeling interactions among multimodal, high-dimensional data is intrinsically challenging due to ultra-high dimensionality and complex dependence structure with high level noise.

By Hongju Park, Zhenyao Ye, Shuo Chen

multimodal safety

arXiv Machine Learning

Jun 3

Spatial Transcriptomics-Guided Alignment Enhances Molecular Profiling in Pathology Foundation Model

arXiv:2606. 03644v1 Announce Type: new Abstract: Comprehensive molecular profiling is essential for modern precision oncology but remains hindered by prohibitive costs, specimen exhaustion, and protracted turnaround times.

By Fengtao Zhou, Yingxue Xu, Zhengyu Zhang, Yihui Wang, Zhengrui Guo, Ling Liang, Jiabo Ma, Cheng Jin, Ziyi Liu, Huajun Zhou, Hongyi Wang, Du Cai, Chenglong Zhao, Xi Wang, Can Yang, Yu Wang, Wenbin Li, Feng Gao, Zhe Wang, Zhenhui Li, Xiuming Zhang, Li Liang, Hao Chen

fine-tuning multimodal safety

arXiv Machine Learning

Jun 3

CoughSense: Five-Class Respiratory Disease Classification via Whisper Encoder Fine-Tuning and Dual-Encoder Cross-Attention Fusion with Balanced Contrastive Learning

arXiv:2606. 02998v1 Announce Type: new Abstract: Automated cough analysis offers a path to low-cost respiratory screening, but most existing work stops at binary COVID-19 detection.

By Nikhil Vincent

fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 3

Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals

arXiv:2606. 02679v1 Announce Type: new Abstract: Multimodal systems often benefit from combining information across language, sound, and visual streams, but this benefit is not guaranteed.

By Jiyuan Liu, Liangwei Nathan Zheng, Wei Emma Zhang, Xinpei Wang, Weitong Chen

multimodal benchmarks

arXiv AI

Jun 3

TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design

arXiv:2605. 20731v2 Announce Type: replace-cross Abstract: Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison.

By Haonan Zhu, Elad Hirsch, Alexandria Minetti, Allison Nulty, Purvanshi Mehta

diffusion multimodal benchmarks safety

arXiv AI

Jun 3

Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models

arXiv:2605. 18160v2 Announce Type: replace-cross Abstract: In recent years, multimodal large language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information.

By Xinpeng Dong, Min Zhang, Kairong Han, Xu Tan, Fei Wu, Kun Kuang

llms computer-vision multimodal benchmarks safety

arXiv AI

Jun 3

Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation

arXiv:2605. 18740v4 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding, where answers often depend on small but decisive evidence in the full image.

By Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, Yaojie Lu

llms agents efficiency multimodal benchmarks

arXiv AI

Jun 3

Learning the Neighborhood: Contrast-Free Multimodal Self-Supervised Molecular Graph Pretraining

arXiv:2509. 22468v2 Announce Type: replace-cross Abstract: High-quality molecular representations are essential for property prediction and molecular design, yet large labeled datasets remain scarce.

By Boshra Ariguib, Mathias Niepert, Andrei Manolache

llms rag diffusion fine-tuning multimodal benchmarks

arXiv AI

Jun 3

TalkPlayData 2: An Agentic Synthetic Data Pipeline for Multimodal Conversational Music Recommendation

arXiv:2509. 09685v5 Announce Type: replace-cross Abstract: We present TalkPlayData 2, a synthetic dataset for multimodal conversational music recommendation generated by an agentic data pipeline.

By Keunwoo Choi, Seungheon Doh, Juhan Nam

llms agents multimodal data-engineering

arXiv AI

Jun 3

CoMPAS3D: A Dataset and Benchmark for Interactive Motion

arXiv:2507. 19684v2 Announce Type: replace-cross Abstract: Socially interactive humanoid robots must engage with humans through their bodies, adapting in real time to a partner's movement, intent, and abilities.

By Bermet Burkanova, Yasaman Etesam, Payam Jome Yazdian, Trinity Evans, Chuxuan Zhang, Zoe Stanley, Paige Tutt\"os\'i, Angelica Lim

llms robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jun 3

PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft

arXiv:2605. 27762v2 Announce Type: replace Abstract: We present PEAM, a Parametric Embodied Agent Memory framework in Minecraft that transforms agent memory from inference-time retrieval into parameter-resident skills internalized through experience.

By Yuchen Guo, Junli Gong, Weicheng Wang, Hongmin Cai, Yiu-ming Cheung, Weifeng Su

llms agents robotics fine-tuning multimodal

arXiv AI

Jun 3

The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios

arXiv:2601. 08173v2 Announce Type: replace Abstract: The rapid evolution of Multi-modal Large Language Models (MLLMs) has advanced workflow automation; however, existing research mainly targets performance upper bounds in static environments, overlooking robustness for stochastic real-world deployment.

By Daocheng Fu, Jianbiao Mei, Rong Wu, Xuemeng Yang, Jia Xu, Ding Wang, Pinlong Cai, Yong Liu, Licheng Wen, Botian Shi

llms agents multimodal benchmarks safety

arXiv AI

Jun 3

Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs

arXiv:2606. 03879v1 Announce Type: cross Abstract: As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled design.

By Wei Ding, Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Yu Wang

llms multimodal benchmarks

arXiv AI

Jun 3

When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics

arXiv:2606. 03569v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated remarkable capabilities but suffer from significant computational overhead during inference.

By Jiahui Wang, Kai Zhang, Mai Han, Huanghe Zhang

llms efficiency multimodal safety

arXiv AI

Jun 3

NeuroArmor: Safe-Variant-Guided Representation Consistency for Selective Re-Anchoring in Jailbreak Defense

arXiv:2606. 03486v1 Announce Type: cross Abstract: Large language models remain vulnerable to jailbreak attacks that hide harmful intent behind seemingly ordinary requests such as role-play, translation, encoding, adversarial suffixes, and multi-turn buildup.

By Zhongyang Lin, Ziran Zhao, Feifei Zhai, Pengyuan Liu

llms multimodal safety

arXiv AI

Jun 3

PhotoCraft: Agentic Reasoning with Hierarchical Self-Evolving Memory for Deep Image Search

arXiv:2606. 03099v1 Announce Type: cross Abstract: Deep Image Search requires multi-step reasoning over rich contextual cues, such as time, location, and event relations.

By Kailin Lyu, Zhiqiang Yuan, Jianwei He, Qiwei Yan, Xuanbo Su, Nanxing Hu, Yang Liu, Ce Hao, Shengqian Qin, Lianyu Hu, Jinchao Zhang, Jie Zhou

llms agents multimodal

arXiv AI

Jun 3

MUSE: A Unified Agentic Harness for MLLMs

arXiv:2606. 03005v1 Announce Type: cross Abstract: Despite rapid progress, multimodal large language models (MLLMs) still fail on tasks that humans solve effortlessly, such as navigating a grid maze from a screenshot or selecting the correct puzzle piece.

By Jianglin Lu, Hailing Wang, Xu Ma, Qihua Dong, Mingyuan Zhang, Yizhou Wang, Yun Fu

llms agents multimodal benchmarks

arXiv AI

Jun 3

AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes

arXiv:2606. 02724v1 Announce Type: cross Abstract: Audio-visual speaker tracking aims to localize and track active speakers by leveraging auditory and visual cues, enabling fine-grained, human-centric scene understanding.

By Yaoting Wang, Yun Zhou, Zipei Zhang, Henghui Ding

computer-vision multimodal benchmarks safety

arXiv AI

Jun 3

CL-DMDF:Dynamic Multimodal Data Fusion Model Based on Contrastive Learning

arXiv:2606. 02659v1 Announce Type: cross Abstract: Multimodal data fusion involves integrating and analyzing information from multiple modalities to uncover latent correlations and complementary patterns, thereby enhancing data processing and decision-making.

By Dong Li, Lingling Zhang, Binghao Han, Linlin Ding, Yue Kou

multimodal

Get the daily briefing