Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,102 stories · RSS feed

arXiv Computation and Language
Sep 11

ReGround: Grounding Reviewer Comments in Multimodal Evidence

ReGround is a new large‑scale dataset that links 10,267 reviewer comments to 16,274 pieces of evidence across 3,656 anonymous scientific papers, addressing the challenge of grounding comments in long multimodal documents. The dataset is constructed by leveraging explicit references in author rebuttals, providing high‑precision annotations. Evaluation shows that simple retrieval over full paper text performs poorly, evidence‑type inference is a major bottleneck, and multimodal evidence offers complementary signals that pure text misses.

By Serwar Basch, Lizhen Qu, Iryna Gurevych
arXiv Machine Learning
Sep 11

Evaluating Time-Series Foundation Models and Multimodal Dietary Context for CGM Forecasting

The study evaluates time‑series foundation models for continuous glucose monitoring (CGM) forecasting across eight public datasets covering Type 1, Type 2, and non‑diabetes populations. Zero‑shot foundation models did not consistently beat strong task‑specific baselines, but lightweight fine‑tuning of models like Chronos‑Bolt improved root‑mean‑square error by up to 18% in both in‑distribution and out‑of‑distribution settings. Incorporating multimodal dietary context via CGMacros and a residual‑based fusion framework further reduced overall RMSE by ~3% and postprandial RMSE by ~15%, indicating that dietary signals add clinically meaningful value beyond CGM alone.

By Bowen Zhang, Hsiu-Wen Cheng, Hongyu Yang, Evie L. Shen, Joleen Vansomphone, Yuna Li, Kerry Zhou, Zitian Qu, Suning Zhao, Xiangning Deng, Hua Zhou, Jin J. Zhou
arXiv Machine Learning
Sep 11

Learning Intrinsic Water-Quality Dynamics with Rainfall for Data-Driven Forecasting

The paper introduces RaiNet, a data‑driven model that jointly learns multiscale water‑quality dynamics and station‑specific rainfall effects. It uses LocTrend to capture irregular water‑quality patterns, constructs station‑oriented rainfall events from gridded precipitation, and applies XGateFusion for lag‑aware fusion across scales. Experiments on three new multimodal datasets show RaiNet surpasses existing time‑series, water‑quality, diffusion‑based, and spatiotemporal models by over 20%, with each module contributing uniquely to performance.

By Ziqi Wang, Hailiang Zhao, Cheng Bao, Daojiang Hu, Wenzhuo Qian, Shuiguang Deng
Hugging Face Trending Papers
Sep 10

The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challenge

The Eloquence team presents three strategies for the Interspeech 2026 MLC‑SLM Task 2, a multilingual MCQA challenge across 21 languages. They fine‑tune Voxtral‑Mini‑3B with LoRA and cross‑lingual augmentations, achieving 0.72 macro‑accuracy; they use multimodal in‑context learning on the frozen Voxtral‑24B to correct label bias, reaching 0.81; and they deploy a training‑free retrieval system with a voice‑anchored memory, scoring 0.68. All approaches surpass the official baseline.

Hugging Face Trending Papers
Sep 10

Combining Synthetic and Real Data for Low-Resource Historical OCR: A Manchu Case Study

The paper investigates how to best combine synthetic and real historical Manchu word images for low‑resource OCR. Using 60,000 synthetic and 20,306 real images, the authors compare three pretrained vision‑language models and a compact CRNN across synthetic‑only, real‑only, joint, and sequential training regimes. Adding real data boosts word accuracy to 95–96%, and ensembling the best recognizers raises it to 98.27% without extra training.

Hugging Face Trending Papers
Sep 10

DINO-Med: A Unified Patch-Based Adaptation Framework for Multi-Modal Medical Image Analysis Applied to Liver Fibrosis Staging

The paper introduces DINO-Med, a patch‑based framework that adapts natural‑image foundation models to multi‑modal medical imaging, specifically for liver fibrosis staging. It processes raw multimodal scans through training‑free registration, automated localization, and mask‑filtered patch extraction, then aggregates patch‑level insights into subject‑level diagnostics. Using the CARE 2025 Liver Track 4 cohort, the DINOv3‑based approach achieved the highest classification accuracy (78.4% for mild fibrosis and 75.8% for cirrhosis) compared to other feature representations.

Hugging Face Trending Papers
Sep 10

2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation

The paper introduces 2AM, a system that keeps task memory solely within a multimodal Agent while using a single RGB‑based, stateless Action Model to execute motions. By compiling interaction history into subtask language and optional 2D grasp/place/move hints, the Agent steers the Action Model, which is trained to tolerate imperfect guidance through dropout, noise, and jitter. On the LIBERO‑Mem benchmark, 2AM achieves 76.3% average completion without depth, geometry, or planners, vastly outperforming the best baseline.

arXiv Machine Learning
Sep 10

SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

arXiv:2609.05533v1 Announce Type: cross Abstract: Long-horizon manipulation is partially observable: the information needed to choose the next action may appear only in observations from minutes earl...

By Cheng Yin, Wang Xu, Junpeng Yang, Sikyuen Tam, Hanyu Liu, Yuan Yao, Xiangrui Zeng, Junbo Cui, Yequan Wang, Zhouping Yin, Yankai Lin
arXiv Machine Learning
Sep 10

Unexplored flaws in multiple-choice VQA make benchmarking unreliable

The paper demonstrates that multiple‑choice visual question answering (MC‑VQA) benchmarks are unreliable because model performance is highly sensitive to semantically neutral prompt formatting choices—such as option ID sets, delimiters, and separators—despite protocols that mitigate option‑order effects. Across seven multimodal large language models and five datasets, the authors observed frequent rank reversals when systematically varying 48 equivalent prompt formats, attributing the instability to tokenizer‑induced token fusion or removal and to how option ID sets influence attention patterns. Consequently, MC‑VQA rankings correlate weakly with open‑ended evaluation, revealing that MC‑VQA reflects option‑selection dynamics as well as multimodal reasoning.

By Fabio Rosenthal, Sebastian Schmidt, Thorsten Graf, Thorsten Bagdonat, Stephan G\"unnemann, Leo Schwinn
arXiv AI
Sep 10

MemForest: Efficient Agent Memory Management via EventTree Partitioning and Progressive Merging

arXiv:2609.08273v1 Announce Type: new Abstract: Agent memory systems have demonstrated significant potential in long-term dialogue, personalized assistants, and video understanding. However, continuo...

By Junxi Wang, Te Sun, Jiayi Zhu, Chen Zhang, Siyuan Li, Xuyang Liu, Zichen Wen, Xiaobing Tu, Jinkui Ren, Xiantao Zhang, Ziqi Yuan, Linfeng Zhang
arXiv AI
Sep 10

Multiple Myeloma Lesion Segmentation on Whole-Body Diffusion-Weighted Imaging via Efficient Anatomical Anticipation and Multimodal Confirmation

arXiv:2609.06165v1 Announce Type: cross Abstract: Whole-body diffusion-weighted imaging (WB-DWI) is widely used for multiple myeloma (MM) assessment, yet automated lesion segmentation remains challen...

By Mengmeng Zhang, Shengqian Huang, Junde Zhou, Xiaoping Wu, Hao Luog, Jing Wanga, Yicheng Sun, Jiao Li, Haibo Zhang, Sheng Xie, Fan Wangg, Qin Wangc, Huadan Xue, Yisheng Lv, Fei-yue Wang