Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,626 stories · RSS feed

arXiv AI

Jun 6

UNIVID: Unified Vision-Language Model for Video Moderation

arXiv:2606. 05748v1 Announce Type: cross Abstract: Global-scale video moderation faces a dual challenge: the need for fine-grained multi-modal reasoning and the demand for interpretable outputs to support downstream enforcement.

By Kejuan Yang, Yizhuo Zhang, Mingyuan Du, Yue Zhang, Dixin Zheng, Kaili Zhao, Yang Xiao, Hanzhong Liang, Kenan Xiao

llms multimodal safety

arXiv AI

Jun 6

EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models

arXiv:2606. 06379v1 Announce Type: cross Abstract: Medical vision-language models (VLMs) have shown increasing potential for clinical image interpretation, including lesion detection and report generation.

By Qiwei Zeng, Hao Wang, Jinghao Lin, Shuchang Ye, Yuezhe Yang, Yige Peng, Haoyuan Che, Jinman Kim, Lei Bi

llms rag multimodal safety

arXiv AI

Jun 6

TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models

arXiv:2606. 06285v1 Announce Type: new Abstract: Time series foundation models (TS-FMs) aim to learn generalizable temporal representations that can be adapted to a wide range of downstream tasks.

By Ziwen Kan, Yishuo Chen, Kecheng Li, Andrew Wen, Xiaomeng Wang, Liwei Wang, Jihao Duan, Song Wang, Hongfang Liu, Tianlong Chen

nlp multimodal benchmarks

arXiv AI

Jun 6

MAviS: A Multimodal Conversational Assistant For Avian Species

arXiv:2603. 07294v2 Announce Type: replace-cross Abstract: Fine-grained understanding and species-specific multimodal question answering are vital for advancing biodiversity conservation and ecological monitoring.

By Yevheniia Kryklyvets, Mohammed Irfan Kurpath, Sahal Shaji Mullappilly, Jinxing Zhou, Fahad Shabzan Khan, Rao Anwer, Salman Khan, Hisham Cholakkal

llms nlp multimodal benchmarks

arXiv AI

Jun 6

MAVEN A Multi-Agent Framework for Multicultural Text-to-Video Generation

arXiv:2605. 16716v4 Announce Type: replace-cross Abstract: Text-to-video (T2V) generation has rapidly progressed in visual fidelity, yet its ability to faithfully represent multiple cultures within a single prompt remains underexplored.

By Shuowei Li, Yuming Zhao, Parth Bhalerao, Oana Ignat

agents diffusion multimodal benchmarks

arXiv Machine Learning

Jun 5

Discrete Causal Representations from Heterogeneous Domains: A Bayesian Approach with Social Survey Applications

arXiv:2606. 06288v1 Announce Type: cross Abstract: Causal representation learning aims to infer the high-level latent causal concepts that give rise to observed low-level measurements.

By Ankur Garg, Michael Stettler, Aaron Schein, Julius von K\"ugelgen

multimodal safety

arXiv Machine Learning

Jun 5

Coreset-Induced Conditional Velocity Flow Matching

arXiv:2605. 12951v2 Announce Type: replace-cross Abstract: We propose Coreset-Induced Conditional Velocity Flow Matching (CCVFM), a generative model that augments hierarchical rectified flow with a data-informed source distribution.

By Xiao Wang, Zihua She, Jianxi Su

diffusion multimodal

arXiv Machine Learning

Jun 5

Harnessing Generalist Agents for Contextualized Time Series

arXiv:2606. 05404v1 Announce Type: cross Abstract: Time series are often embedded in rich contexts that are essential for holistic modeling.

By Zihao Li, Kaifeng Jin, Yuanchen Bei, Jiaru Zou, Avaneesh Kumar, Xuying Ning, Yanjun Zhao, Mengting Ai, Baoyu Jing, Hanghang Tong, Jingrui He

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 5

HANDOFF: Humanoid Agentic Task-Space Whole-Body Control via Distilled Complementary Teachers

arXiv:2606. 06493v1 Announce Type: cross Abstract: For a humanoid robot to be deployed in the real world, the choice of command space (i.

By Lizhi Yang, Junheng Li, Nehar Poddar, Yiling Hou, Gio Huh, Robert Griffin, Georgia Gkioxari, Aaron Ames

agents robotics fine-tuning efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 5

Scaling Laws for Behavioral Foundation Models over User Event Sequences

arXiv:2606. 05257v1 Announce Type: new Abstract: Foundation models are increasingly trained on sequences of user actions in recommendation, payments, fraud, and commerce, but these models still lack the kind of compute calibration that scaling laws provide for language models.

By Rickard Br\"uel Gabrielsson

llms multimodal benchmarks

arXiv Machine Learning

Jun 5

Steering Vectors are an Adversarial Attack Surface

arXiv:2606. 05958v1 Announce Type: new Abstract: Activation steering has become a popular way to control Large Language Model (LLM) behavior without fine-tuning.

By Abzal Aidakhmetov, Donato Crisostomi, Tommaso Mencattini, Adrian Robert Minut, Iacopo Masi, Emanuele Rodol\`a

llms fine-tuning multimodal safety

arXiv Machine Learning

Jun 5

GIPO: Gaussian Importance Sampling Policy Optimization

arXiv:2603. 03955v2 Announce Type: replace Abstract: Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation.

By Chengxuan Lu, Zhenquan Zhang, Shukuan Wang, Qunzhi Lin, Yanjie Li, Baigui Sun, Yang Liu

agents reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 5

Learning Self-Correction in Vision-Language Models via Rollout Augmentation

arXiv:2602. 08503v2 Announce Type: replace-cross Abstract: Self-correction is essential for solving complex reasoning problems in vision-language models (VLMs).

By Yi Ding, Ziliang Qiu, Bolian Li, Ruqi Zhang

llms reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 5

Step-adaptive multimodal fusion network with multi-scale cloud feature learning for ultra-short-term solar irradiance forecasting

arXiv:2606. 06102v1 Announce Type: cross Abstract: Ultra-short-term solar irradiance prediction is critical for photovoltaic system dispatch and power grid stability.

By Jingxin Zhang Xiaoqin Wang

multimodal benchmarks

arXiv Machine Learning

Jun 5

DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models

arXiv:2606. 05758v1 Announce Type: cross Abstract: Many modern vision-language models (VLMs) build on autoregressive decoding of discrete tokens.

By Zhuoming Liu, Jinhong Lin, Kwan Man Cheng, Lin Zhang, Shayok Bagchi, Yin Li

llms diffusion robotics multimodal

arXiv Machine Learning

Jun 5

Equivariant Neural Belief Propagation

arXiv:2606. 06344v1 Announce Type: new Abstract: Probabilistic inference over spatially embedded variables requires beliefs that respect $SE(3)$ symmetry, yet existing equivariant networks produce only scalars and vectors -- not the rank-2 precision tensors needed for anisotropic uncertainty, and single-component messages collapse multi-modal energy landscapes to physically meaningless averages.

By Zehua Cheng, Wei Dai, Jiahao Sun

agents diffusion robotics multimodal

arXiv Machine Learning

Jun 5

LEVANTE-bench: Multi-Scale Comparison of VLMs to Children Using Cognitive Tasks (or, "Is Your VLM Smarter Than a 5th Grader?")

arXiv:2606. 05497v1 Announce Type: new Abstract: Given the inherently multimodal nature of human experience, vision-language models (VLMs) hold substantial promise for modeling human cognition as it grows and develops with experience.

By Alvin Wei Ming Tan, David Cardinal, Tania Lorido-Botran, Laura Bravo-Sanchez, Sunny Yu, Michael C. Frank

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 5

A Survey on Diffusion Language Models

arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.

By Tianyi Li, Mingda Chen, Bowei Guo, Zhiqiang Shen

llms diffusion nlp efficiency multimodal benchmarks

arXiv Machine Learning

Jun 5

SlotGCG: Exploiting the Positional Vulnerability in LLMs for Jailbreak Attacks

arXiv:2606. 05609v1 Announce Type: cross Abstract: As large language models (LLMs) are widely deployed, identifying their vulnerability through jailbreak attacks becomes increasingly critical.

By Seungwon Jeong, Jiwoo Jeong, Hyeonjin Kim, Yunseok Lee, Woojin Lee

llms multimodal safety

arXiv Machine Learning

Jun 5

Electricity price forecasting across Norway's five bidding zones in the post-crisis era

arXiv:2604. 26634v2 Announce Type: replace Abstract: Norway's electricity market is heavily dominated by hydropower, but the 2021-2022 energy crisis and stronger integration with Continental Europe have fundamentally altered price formation, reducing the reliability of forecasting models calibrated on historical data.

By My Thi Diem Phan, Trung Tuyen Truong, Hoai Phuong Ha, Dat Thanh Nguyen

multimodal benchmarks safety

Get the daily briefing