Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,624 stories · RSS feed

arXiv AI
Aug 25

MetaCaster: Meta-Harness-Optimized Agent for End-to-End Few-Shot Learning of Lightweight Time Series Forecasters

MetaCaster is a meta-harness-optimized multi-agent framework that enables few-shot learning for lightweight time series forecasters. It uses agentic data generation to automatically train specialized forecasters from only a few examples and textual contexts, positioning agents as intermediary engineers rather than direct forecasters. Experiments on 18 datasets and 23 lightweight forecasters show that MetaCaster achieves data and computational efficiency while maintaining high forecasting quality.

By ChengAo Shen, Wenchao Yu, Fangyu Wu, Dongjin Song, Hanghang Tong, Dongsheng Luo, Wei Cheng, Haifeng Chen, Jingchao Ni
arXiv AI
Aug 25

CallScreenBench: Benchmarking Small Language Models as Phone Secretaries

CallScreenBench is a benchmark for evaluating small, on-device language models that act as phone secretaries, focusing on their ability to handle unknown inbound calls without a cooperative task. The benchmark measures owner endorsement through five call-and-note metrics, each paired with counter-metrics and uncertainty estimates, and includes guardedness diagnostics to identify safe, tool‑free proxies. Results across 4‑bit checkpoints of 0.6‑4 B parameter models show varying performance on service, recall, plausibility, and triage discrimination, highlighting trade‑offs between quality and guardedness.

By Jiaqi Gan, Haoyuan Tang, Jamey Z. Liang, Siying Chen, Ankit Raj, Kidus Zewde, Yuchen Zhou, Yuxin Zhang, Simiao Ren
arXiv Computer Vision
Aug 25

Adapting Dense Vision-Language Relationships for Multi-label Classification with Partial Label

The paper introduces Language-driven Dense Semantic Adaptor (LDSA) for multi-label image classification with incomplete annotations. LDSA leverages multimodal pretrained CLIP models to extract prior-adaptive relationships, employing a densely contrastive adaptor for visual contrastive constraints and a language-driven interactive decoder with class-specific prompt tuning. Experiments show LDSA achieves state‑of‑the‑art performance on public benchmarks and reveals implicit semantic relationships through its learning scheme.

By Cheng Chen, Yifan Zhao, Jia Li
arXiv Computer Vision
Aug 25

TONAV: Task-Oriented Navigation and Action-Velocity Chunk Learning for Articulated Object Quadrupedal Mobile Manipulation

arXiv:2608.22296v1 Announce Type: cross Abstract: Quadruped mobile manipulation requires two tightly coupled capabilities: reaching manipulation-ready configurations and maintaining stable contact th...

By Haoran Lin, Mingyu Yang, Pengfei Qi, Kehan Chen, Qiang Diao, Liangji Zeng, Wenrui Chen, Yaonan Wang, Kailun Yang
arXiv Computer Vision
Aug 25

Investigating Relational Reasoning in VLMs

arXiv:2608.23518v1 Announce Type: new Abstract: Vision-Language Models (VLMs) achieve strong performance in visual reasoning tasks, but it remains unclear whether they understand visual relations, or...

By Adhithya Laxman Ravi Shankar Geetha, Aulia Kharis Rakhmasari, Haleema Ramzan, Xander Yap
arXiv AI
Aug 25

ExtrinSplat: Decoupling Geometry and Semantics for Open-Vocabulary Understanding in 3D Gaussian Splatting

ExtrinSplat is a new framework that separates geometry from semantics in 3D Gaussian Splatting scenes. It clusters Gaussians into overlapping 3D object groups and uses a Vision‑Language Model to generate lightweight textual hypotheses, creating an extrinsic index layer that handles complex polysemy. This approach reduces adaptation time from hours to minutes, cuts storage overhead by orders of magnitude, and outperforms existing embedding‑based methods on open‑vocabulary 3D object selection and semantic segmentation benchmarks.

By Jiayu Ding, Xinpeng Liu, Zhiyi Pan, Shiqiang Long, Ge Li
arXiv Computer Vision
Aug 25

EchoWM: Open and Enterable Omnimodal World Models

arXiv:2608.23189v1 Announce Type: new Abstract: We present EchoWM, an omnimodal world model for enterable generative media that responds to continuous navigation while jointly generating 720p video,...

By Songchun Zhang, Yaowei Li, Junhao Zhuang, Weiyang Jin, Haoyu Wang, Xin Lu, Yilang Sun, Shiyi Zhang, Haoran Li, Xiaoxiao Ma, Yuming Li, Yijun Liu, Yaofeng Su, Yanwen Ma, Haoyu Wu, Zihan Su, Yue Ma, Lvmin Zhang, Haoyang Huang, Zeyue Xue, Anyi Rao, Nan Duan
arXiv AI
Aug 25

Mitigating Sample-Level Imbalance via Probabilistic Separation for Adaptive Multimodal Fusion

The paper introduces a framework to tackle modality imbalance in multimodal learning by focusing on sample-level variations. It defines a Modality Gap metric to measure prediction discrepancies, models the resulting bimodal distribution with a Gaussian Mixture Model, and uses Bayesian probabilities for soft separation of balanced and imbalanced samples. A two‑stage training process—Warm‑up and Adaptive Training—reallocates loss weights based on the GMM, strengthening alignment for imbalanced samples while favoring fusion for balanced ones, and shows superior performance over existing baselines.

By Zhiwen Yu, Zhaocheng Liu, Xiaoqing Liu, Huanqiang Zeng, C. L. Philip Chen
arXiv AI
Aug 25

Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments

The paper introduces two minimal simulation foundations—SD-AgentFoundry-2D and SD-AgentFoundry-3D—for educational and rapid prototyping use with large language models (LLMs) and vision-language models (VLMs). SD-AgentFoundry-2D offers a 2D multi‑agent environment where LLM agents move, communicate, and react to local events such as fire, while SD-AgentFoundry-3D provides a 3D digital‑twin setting where a VLM interprets first‑person images to generate natural‑language movement instructions. Both frameworks run locally on macOS, Windows, and Linux, are intentionally lightweight, and are open for modification rather than being finished applications.

By Ryuki Hyodo
arXiv AI
Aug 25

Act with Intent: Distilling Behavior Intent for Vision-Language-Action Models

The paper introduces Intention Distillation (INDI), a method that injects behavior-level intent into Vision‑Language‑Action (VLA) model decoders by leveraging a frozen teacher vision‑language model to interpret demonstrations. During training, the teacher processes the current observation, instruction, coarse action summary, and execution video, producing a multimodal intent representation that the VLA decoder uses alongside trajectory and execution features to predict actions. Experiments on SimplerEnv‑Bridge, RoboCasa Kitchen, and real‑world tasks show that INDI consistently improves success rates, especially on longer‑horizon tasks, demonstrating that explicit modeling of semantic intent benefits action decoders.

By Sangoh Lee, Sangwoo Mo, Wook-Shin Han
arXiv Computer Vision
Aug 25

Learning Sample-wise Rank-aware Interpolation Weights for Composed Visual Data Retrieval

The paper introduces SRAIN, a framework that learns sample‑wise, rank‑aware interpolation weights for composed visual data retrieval. Instead of relying on complex multimodal large language models, SRAIN uses simple linear interpolation in embedding space, dynamically predicting query‑specific weights through batch‑wise rank‑aware estimation and a compact memory bank for hard negatives. This approach achieves state‑of‑the‑art performance on composed video retrieval and competitive results on composed image retrieval while significantly reducing query‑time latency.

By Boseung Jeong, Taegyu Park, Donghyeon Kwon, Hyunsouk Cho, Suha Kwak