Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,102 stories · RSS feed

arXiv Machine Learning
Sep 14

Efficient Vision-Language-Action Management and Serving for Robot Factories

Robion is a new serving and management system designed to run Vision‑Language‑Action (VLA) models on multi‑GPU edge servers for robot factories. It splits the VLM and ADiT stages within a single GPU, shares streams across multiple models, and prioritizes requests by remaining SLO time, enabling high robot load while meeting strict latency requirements. In experiments, Robion achieves 6.7× higher robot load than vLLM‑Omni and 1.5× higher than a monolithic pipeline, and can serve 64 robots on a 4‑GPU server with 98% SLO attainment.

By Dionysios Adamopoulos, Nattapol Chanpaisit, Basel Fakhri, Christina Giannoula
arXiv Computer Vision
Sep 14

ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoning

ChitraMiti introduces a synthetic benchmark of 12,874 Bengali planar geometry problems with structured 15‑attribute descriptions, alongside a complementary set of 500 textbook diagrams. Using a three‑phase protocol that tests diagram‑only, diagram‑plus‑description, and description‑only inputs, the study finds that description‑only performance matches diagram‑plus‑description performance across several VLMs, yet models still struggle with cross‑modal verification. Fine‑tuning on ChitraMiti improves results on both datasets, though a gap remains compared to the best zero‑shot model.

By Khan Raiyan Ibne Reza, Sanjana Aktar Maria, Sumaiya Tabassum Nimi, Md Adnan Arefeen
arXiv Computer Vision
Sep 14

Physics-Aware Video Generation via Agentic Planning and Graph-Guided Optimization

PhysPlan is a training‑free guidance framework that enhances video diffusion models by incorporating physical awareness through agentic physics simulation. It uses a vision‑language model to generate a Chain‑of‑Visual‑Thought representation of kinematic trajectories and 3D depth, which then drives an object‑centric test‑time optimization that isolates kinematic changes and locks the passive environment. The framework also employs Kinetic Intensity Profiling to adapt hyperparameters to varying physical deformations, and demonstrates superior performance on PhyGenBench and Physics‑IQ benchmarks compared to existing VDM baselines.

By Minh-Loi Nguyen, Xuan-Vu Le, Thanh-Toan Do, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le
arXiv Computer Vision
Sep 14

DementiaCare-Bench: A Modality-Validated Video Benchmark

DementiaCare-Bench is a new video benchmark designed to evaluate video-language models (VLMs) on understanding behavioral and psychological symptoms of dementia (BPSD). It contains 56 caregiver training videos, 94 clips across nine BPSD categories, and 2023 questions that are grounded in transcript spans and labeled by their visual demand. The benchmark reveals that most VLMs perform well on questions answerable by language alone but struggle with those requiring ordered frames, and a lightweight LoRA fine‑tune (DemCare‑VLM) can improve video dependence.

By Afrouz Sheikholeslami, Yuankai Qi, Xuyun Zhang, Luping Zhou, Amin Beheshti, Quan Z. Sheng, Ming-Hsuan Yang
arXiv Machine Learning
Sep 14

FRIST: FMRI Representation Informed Shared-space Training Improves EEG-only Individual-Finger BCI Decoding

The paper introduces FRIST, a two‑stage EEG decoding framework that uses fMRI data to learn spectral projections and class geometry, then refines EEG predictions. In experiments with 12 participants, FRIST improves finger‑level motor decoding accuracy across both movement execution and motor imagery tasks, outperforming EEG‑only baselines and generalizing across different EEG backbones. The method demonstrates that fMRI’s spatial resolution can enhance noninvasive EEG‑based BCI performance.

By Jintao Zhang, Yidan Ding, Joshua Kosnoff, Maxim Karrenbach, Hanwen Wang, Bin He
arXiv Computer Vision
Sep 14

AnchorVLN: Geometry-Anchored Vision-Language Grounding Reasoning for Open-Vocabulary Navigation

AnchorVLN is an open‑vocabulary vision‑language navigation system that separates semantic proposals from geometric metrics. It uses a VLM to generate semantics while a geometry module supplies reliable metric quantities such as range and bearing, all within a Model Context Protocol server. The system achieves 64.4% on instruction following and improves object‑reference accuracy, reducing median center error from 3.37 m to 2.48 m.

By Long Giang Vu, Chengkai Yao, Yuxin Liu, FNU Aryan, Rajath Chandrashekar Aralikatti
arXiv Computer Vision
Sep 14

A Dual Cross-Attention Framework for Colposcopic CIN Grading and Swede Score Prediction Using a New Multi-Center Dataset

The paper introduces a dual cross‑attention deep learning framework for automated grading of Cervical Intraepithelial Neoplasia (CIN) and prediction of Swede scores, using a newly released BUET Multi‑Center Colposcopy Dataset. The architecture fuses paired multimodal cervigrams and employs a custom composite loss to handle class imbalance, achieving 71.85% accuracy and 86.23% AUC‑ROC for three‑class CIN grading, and AUC‑ROC values between 75.7% and 88.4% for individual Swede score components. The total predicted Swede Score has a mean absolute error of 1.489, indicating potential for AI‑assisted colposcopy screening in resource‑limited settings.

By Dania Khan, Nuzhat Aisha Shaikh, Asfina Hassan Juicy, Raiyun Kabir, S M Shahida, Taufiq Hasan
arXiv Computation and Language
Sep 14

Quantifying Consonant Contributions to Word Intelligibility via Acoustic Masking

The study introduces a scalable acoustic‑masking method to quantify how much each consonant contributes to word intelligibility. By silencing individual consonants in isolated words and measuring misrecognition rates with three ASR models, the authors define a mask‑induced misrecognition rate (MMR). Across English, Spanish, German, and Czech, MMR negatively correlates with phoneme frequency and positively with functional load, revealing that consonant importance varies by language.

By Eunjung Yeo, Kwanghee Choi, Krupaben Kothadia, Visar Berisha, Julie M. Liss, David R. Mortensen, David Harwath
arXiv Machine Learning
Sep 14

ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts

ExpertHTR is a unified vision‑language framework for handwritten text recognition that tackles the challenge of small, heterogeneous datasets by organizing structural annotations into a common Page‑Region‑Line representation. It defines four related training tasks—complete transcription, physical‑line coverage, text localization, and localized recognition—without extra manual labels. The model combines a jointly trained dense backbone with a sparse Mixture‑of‑Experts architecture, using Sparsegen routing and regularization to adaptively activate experts, achieving state‑of‑the‑art results on the IAM benchmark and outperforming general‑purpose OCR systems on most datasets.

By Dang Hoai Nam, Nguyen Duy Hieu, Quang Huu Hieu, Vo Nguyen Le Duy
arXiv Computer Vision
Sep 14

UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interaction

arXiv:2609.12898v1 Announce Type: new Abstract: Fine-grained robotic manipulation depends on understanding parts, not only whole objects. Existing 3D foundation models tend to be either generalized b...

By Xinqiang Yu, Zekun qi, Jiawei He, Wenyao Zhang, Xuchuan Chen, Guaocai Yao, Li Yi, Zhaoxiang Zhang, He Wang
arXiv Machine Learning
Sep 14

Inferring Dislocation Microstructures from X-ray Diffraction via Cross-Modal Contrastive Learning

The paper presents a cross‑modal learning framework that predicts three‑dimensional dislocation density fields directly from X‑ray diffraction data. By pairing simulated dislocation density fields with virtual diffraction patterns and embedding them into a shared 2‑D latent space via contrastive learning, the authors achieve strong alignment between structural and diffraction representations. Experiments show that model performance improves rapidly with dataset size, reaching near‑saturation with about 500 representative observations out of 10,000, and the predicted structures capture the dominant spatial features of the underlying microstructures.

By Benjamin Udofia, Nicolas Bertin, Markus Stricker
arXiv Machine Learning
Sep 14

Scaling Online Complex Event Detection with Synthetic Supervision and Mamba-Based Neural Algorithmic Reasoning

The paper presents NAROCE, a Neural Algorithmic Reasoning framework for online complex event detection (CED). It decouples rule learning from sensor semantics by pretraining a Mamba-based rule reasoner on synthetic atomic event traces and then adapting it to raw sensor inputs with limited labeled data. Experiments on a simulator‑generated benchmark show that NAROCE matches or surpasses strong baselines while using far fewer labeled sequences and computational resources.

By Liying Han, Gaofeng Dong, Xiaomin Ouyang, Kang Yang, Lance Kaplan, Federico Cerutti, Mani Srivastava
arXiv Computer Vision
Sep 14

Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimaging

The paper introduces Neuro‑JEPA, a sparse multimodal foundation model that learns unified representations of brain MRI across T1w, T2w, and FLAIR sequences using a latent predictive objective and a Mixture‑of‑Experts architecture. It was pretrained on over 1.5 million scans from 428,647 studies and systematically evaluates architectural, masking, objective, and sparsity choices for robust multimodal representation learning. Across 47 tasks from three health systems and 12 public datasets, Neuro‑JEPA consistently outperforms a simple CNN baseline, demonstrating its effectiveness for both clinical and research applications.

By Haoxu Huang, Long Chen, Jingyun Chen, Jinu Hyun, James Ryan Loftus, Kara Melmed, Daniel Orringer, Jennifer Frontera, Seena Dehkharghani, Arjun Masurkar, Narges Razavian