Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

7,971 stories · RSS feed

arXiv AI

Oct 1

ActionGuard: Tool Call Authorization under Poisoned Skills

arXiv:2609.39450v1 Announce Type: cross Abstract: LLM-based agents extend their capabilities through third-party skills that provide task-specific instructions, scripts, and tool-use procedures. Howe...

By Jihun Han, Yejin Jang, Byung Il Kwak, Mee Lan Han

llms agents multimodal

arXiv AI

Oct 1

Less Data, Better Timing: Student-Curriculum Coupling for VLM On-Policy Distillation in Temporal Video Grounding

arXiv:2609.40055v1 Announce Type: cross Abstract: On-policy distillation (OPD) provides dense supervision directly on student-generated trajectories, making it an effective post-training strategy for...

By Jiacheng Qiu, Yunsoo Kim, Ruichen Xu, Jian Luo, Petar M. Djuri\'c, Sima Mofakham

llms efficiency multimodal benchmarks

arXiv AI

Oct 1

ChronoGraph: Functional 4D Scene Graphs with Vision-Language Models for Interaction Understanding and Grounded Planning

arXiv:2609.39665v1 Announce Type: new Abstract: Embodied agents must determine where to act, anticipate the resulting scene changes, and interpret observed outcomes to guide subsequent actions. This...

By Chenyangguang Zhang, Malgorzata Gwiazda, Guanlong Jiao, Yuanchen Ju, Federico Tombari, Koushil Sreenath, Marc Pollefeys, Sunghwan Hong

llms agents reinforcement-learning robotics fine-tuning multimodal

arXiv AI

Oct 1

CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models

arXiv:2609.38810v1 Announce Type: cross Abstract: As vision language models are increasingly deployed in clinical diagnosis, understanding how they internally resolve competing visual and textual sig...

By Chunzheng Zhu, Jiaqi Zeng, Hongbo Zhao, Yihang Chen, Yijun Wang, Jianxin Lin

llms multimodal benchmarks safety

arXiv AI

Oct 1

Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models

arXiv:2605.07872v2 Announce Type: replace-cross Abstract: Multimodal reward models have advanced substantially in text and image domains, yet progress in video understanding reward modeling remains s...

By Yuancheng Wei, Linli Yao, Lei Li, Haojie Zhang, Hao Zhou, Fandong Meng, Xu Sun

reinforcement-learning multimodal benchmarks

arXiv AI

Oct 1

VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes

arXiv:2605.26380v2 Announce Type: replace-cross Abstract: Frontier multimodal large language models (MLLMs) have been reported to achieve over 90\% accuracy on fine-grained perception benchmarks. How...

By Jingru Chen, Yiming Liu, Mingtao Chen, Sijie Chen, Richeng Xuan, Liang Yang, Zhichao Hu, Fanyang Lu

llms multimodal benchmarks

arXiv AI

Oct 1

SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models

arXiv:2609.38716v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have made remarkable progress across visual perception tasks, yet spatial reasoning remains a persistent weaknes...

By Rafi Ibn Sultan, Xiangyu Zhou, Md. Sajid Alam Chowdhury, Chengyin Li, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu

llms multimodal benchmarks

arXiv Computer Vision

Oct 1

Exploiting Vulnerabilities: Universal Adversarial Attacks on Vision-Language-Action Models in Robotics

arXiv:2609.39178v1 Announce Type: cross Abstract: Recently, Vision-Language-Action (VLA) models have revolutionized robotic manipulation by seamlessly integrating visual perception, language understa...

By Songhua Yang, Ziyu Liu, Yuanwei Liu, Xuetao Li, Xuanye Fei, He Huang, Zheng Wang, Miao Li

robotics multimodal safety

arXiv AI

Oct 1

Improving OCR Faithfulness via Gated and Attenuated On-Policy Distillation

arXiv:2609.38282v1 Announce Type: new Abstract: Vision-language models may rewrite anomalous text in images into linguistically plausible expressions, compromising OCR transcription faithfulness. Seq...

By Baode Wang, Zuming Huang, Kexuan Ren, Jun Huang, Wei Chu

llms computer-vision reinforcement-learning efficiency multimodal

arXiv AI

Oct 1

Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation

arXiv:2609.39168v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet existing...

By Zhihan Zhang, Lizi Liao

llms reinforcement-learning multimodal benchmarks

arXiv AI

Oct 1

Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Models

arXiv:2609.38465v1 Announce Type: cross Abstract: Unified multimodal models (UMMs) are increasingly designed around gradient conflict between understanding and generation objectives. The premise that...

By Shuyang Jiang, Fucheng Deng, Yuchuan Luo, Zhenyu Wu

multimodal

arXiv AI

Oct 1

SceneJail: Exploiting Video Scenario Context to Jailbreak Multimodal LLMs

arXiv:2609.38899v1 Announce Type: cross Abstract: Video Multimodal Large Language Models (Video-MLLMs) support reasoning over video inputs, yet remain vulnerable to jailbreak attacks that elicit poli...

By Wenyu Chen, Li Wang, Chuanchao Zang, Xiangtao Meng, Xinyu Gao, Jianing Wang, Zheng Li, Shanqing Guo

llms multimodal safety

arXiv AI

Oct 1

ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation

arXiv:2609.39306v1 Announce Type: cross Abstract: Iterative self-distillation enables LLM agents to learn from successive deployments, offering a path toward recursive self-improvement (RSI). Yet our...

By Shengjie Jin, Hengbo Xu, Zelong Sun, YuJie Guo, Zhiwu Lu

llms agents efficiency multimodal

arXiv AI

Oct 1

CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding

arXiv:2609.39924v1 Announce Type: cross Abstract: Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, m...

By Yulong Liu, Xiaotian Han, Junyuan Shang, Yuchen Ding, Zhenyu Zhang, Shuohuan Wang, Guibo Zhu, Sirui Han, Dianhai Yu

llms rag multimodal benchmarks

arXiv AI

Oct 1

LongEmo: Towards Emotion Understanding and Reasoning in Long Videos

arXiv:2609.40079v1 Announce Type: cross Abstract: While recent Multimodal Large Language Models (MLLMs) have shown promise in affective computing, their reasoning capabilities are largely confined to...

By Shuo Zhang, Yifan Zhou, Han Wang, Jinsong Zhang, Jingyu Li, Hongbing Li, Zhejun Zhang, Chengyi Zhao, Yuquan Hao, Yitong Liu, Jiyin Li, Ruiqi Tang, Zixuan Lin, Yi Luo, Xurui Zhang, Ronghao Chen, Huacan Wang, Lei Li

llms agents multimodal benchmarks

arXiv AI

Oct 1

MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

arXiv:2609.40195v1 Announce Type: cross Abstract: Long-term egocentric video enables personalized AI assistants to reason about daily life. However, as video histories grow to hundreds of hours spann...

By Guangzhi Xiong, Xinyuan Zhang, Xiao Yang, Hyokun Yun, Kai Zhang, Shiun-Zu Kuo, Hyeonjeong Ha, Xilun Chen, Kai Sun, Lucas Liang, Guangqiang Dong, Ejaz Ahmed, Ahmed A Aly, Anuj Kumar, Raffay Hamid, Aidong Zhang, Xin Luna Dong

agents reinforcement-learning multimodal benchmarks

arXiv AI

Oct 1

From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature

arXiv:2511.22232v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are increasingly capable in medical imaging, yet most focus on single-image settings. Clinical inter...

By Zhen Chen, Yihang Fu, Rong Zhou, Serina Applebaum, Min Kyu Kim, Aidan Gilson, Morten Lee, Salahudeen Mirza, Gabriel Madera, Mauro Giuffre, Yuanting Pan, Roy Jiang, Hyunjae Kim, Hua Xu, Qingyu Chen

llms reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Oct 1

ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining

arXiv:2603.02767v4 Announce Type: replace-cross Abstract: Image--text contrastive pretraining has become a dominant paradigm for visual representation learning, yet existing methods often yield repre...

By Hanpeng Liu, Zidan Wang, Shuoxi Zhang, Zonglin Zhao, Zihao Bo, Rinyoichi Takezoe, Kaiwen Long, Yaqian Li, Kun He

multimodal benchmarks safety

arXiv AI

Oct 1

Language-Conditioned World Modeling for Visual Navigation

arXiv:2603.26741v2 Announce Type: replace-cross Abstract: Goal-conditioned visual navigation has been a long-standing testbed for embodied AI. We study a natural language-conditioned variant, languag...

By Yifei Dong, Fengyi Wu, Yilong Dai, Lingdong Kong, Guangyu Chen, Yetong Sha, Qiyu Hu, Feng Liu, Siyu Huang, Qi Dai, Zhi-Qi Cheng

agents diffusion robotics multimodal benchmarks

arXiv AI

Oct 1

Grounded World Model: Latent Planning with Language Goals

arXiv:2604.11751v2 Announce Type: replace-cross Abstract: World models such as DINO-WM and LeWM specify the goal with an image, which is difficult to obtain in advance for novel tasks. We present the...

By Quanyi Li, Lan Feng, Haonan Zhang, Wuyang Li, Letian Wang, Alexandre Alahi, Harold Soh

rag robotics fine-tuning multimodal

Get the daily briefing