Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,102 stories · RSS feed

arXiv Computation and Language
Sep 10

Qwen-Audio-3.0-ASR Technical Report

The Qwen-Audio-3.0-ASR Technical Report introduces a Mixture-of-Experts large language model-based automatic speech recognition system that addresses real‑world production challenges such as regional dialects, dynamic entities, hotwords, long‑range context, and disfluent speech. Built on the Qwen backbone and trained on tens of millions of hours of speech data, it supports transcription in 30 languages and 16 Chinese dialects, and offers industry‑domain entity recognition, hierarchical hotword customization, single‑pass polishing, and long‑audio contextual modeling. A streaming variant, Qwen-Audio-3.0-ASR-Streaming, is also presented for low‑latency applications, with evaluations showing state‑of‑the‑art performance against leading commercial systems.

By Chuanmeng Bian, Daren Chen, Peixin Chen, Zhigao Chen, Zhiyun Fan, Zhifu Gao, Bo Gong, Qing Gu, Jiajun He, Yawei Hu, Yunjie Ji, Jingbei Li, Xiangang Li, Xu Li, Zengxi Li, Zheng Li, Chengdong Liang, Baiji Liu, Ying Liu, Bin Ma, Yiping Peng, Yuezhang Peng, Zhendong Peng, Yu Pu, Yang Shi, Xin Shu, Jian Tang, Biao Tian, Peiyao Wang, Tianzi Wang, Wen Wang, Wupeng Wang, Cheng Wen, Yuzhong Wu, Zijian Xia, Yunchong Xiao, Nan Yang, Jianwei Yu, Jixing Yu, Binbin Zhang, Lei Zhang, Sitong Zhao, Guangdong Zhou, Yuan Zhou, Jianheng Zhuo
arXiv Machine Learning
Sep 10

Generalizing HVAC Control With Domain Randomized Reinforcement Learning

The paper introduces NOMAD‑RL, a reinforcement learning controller for HVAC systems that learns to adapt across diverse thermal zones via a universal thermostat interface. It employs an adaptive domain randomization scheme using physics‑informed normalizing flows to generate realistic, multimodal training data, enabling the recurrent policy to handle partial observability. Experiments show NOMAD‑RL outperforms constant‑setpoint PID and non‑randomized RL, and rivals well‑tuned model predictive control, especially in multi‑zone scenarios.

By Pablo Boitel, Kun Zhang
arXiv Machine Learning
Sep 10

A Specialized Large Multimodal Model for Interpreting PET/CT in Head and Neck Cancer

A specialized large multimodal model, LLaVA‑NeXT, was fine‑tuned on a curated two‑level curriculum of PET/CT image‑conversation pairs to interpret head and neck cancer scans. In external validation across four institutions, the model achieved high ROUGE and similarity scores and outperformed generalist models such as ChatGPT, with primary tumor classification accuracy of 83.14% internally and 69.03% externally. The study demonstrates that domain‑specific LMMs can provide fast, accurate diagnostic support for PET/CT imaging.

By Haengbok Chung, SunGyu Kim, Joo hyun Lee, Sangjin Bae, Min Jeong Cho, Minseok Suh, Jae Sung Lee
arXiv Machine Learning
Sep 10

Cross-modal learning for SAR target recognition using optical vision foundation models

The paper proposes a cross‑modal framework that uses a frozen DINOv3 optical vision foundation model to create class‑level prototypes for Synthetic Aperture Radar (SAR) target recognition. By aligning SAR embeddings to these optical prototypes, the SAR model learns to classify SAR images without needing paired optical data. Experiments on the UNICORNv2 dataset show that this prototype alignment improves SAR classification accuracy compared to baseline methods and yields clearer class separation in the embedding space.

By Lucas Hirsch, James R. Hopgood, Javid Khan, Yoann Altmann, Mike E. Davies
arXiv Machine Learning
Sep 10

To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models

The paper introduces selective adaptation for vision‑language models, questioning whether test‑time adaptation (TTA) should always be applied. By analyzing per‑sample predictions before and after adaptation, the authors find that many adaptations are negligible or even harmful, flipping correct predictions. They propose Cross‑Augmentation Similarity (CAS), which skips adaptation when predictions across augmented views are highly similar, achieving comparable or better accuracy while reducing adaptation by up to 85%.

By Siru Jiang, Yuwei Liang, Jian Liang, Ran He, Tieniu Tan
arXiv Computation and Language
Sep 10

X2-NativeCursor: Native-Token Text Progress Tracking for Incremental-Text Streaming Codec TTS

X2-NativeCursor is a lightweight observer that tracks text progress in incremental‑text streaming TTS by aligning native speech tokens to the original text before waveform decoding. It uses a normalization plan and a local matcher to estimate the current label position, converting these estimates into a non‑backtracking cursor. The method achieves a mean absolute error of 0.151 Chinese characters with 80‑ms lookahead, outperforming a waveform‑based baseline and reducing alignment real‑time factor from 0.3598 to 0.0180.

By Zehan Liu, Carl Chen, Rime Wen, Kaiqi Fu, Altman Lin, Shawn Qin, Lights Shi, Roy Gan, Hao Wang, Qian Wang
arXiv AI
Sep 10

Multiple Myeloma Lesion Segmentation on Whole-Body Diffusion-Weighted Imaging via Efficient Anatomical Anticipation and Multimodal Confirmation

arXiv:2609.06165v1 Announce Type: cross Abstract: Whole-body diffusion-weighted imaging (WB-DWI) is widely used for multiple myeloma (MM) assessment, yet automated lesion segmentation remains challen...

By Mengmeng Zhang, Shengqian Huang, Junde Zhou, Xiaoping Wu, Hao Luog, Jing Wanga, Yicheng Sun, Jiao Li, Haibo Zhang, Sheng Xie, Fan Wangg, Qin Wangc, Huadan Xue, Yisheng Lv, Fei-yue Wang
arXiv Machine Learning
Sep 10

Unexplored flaws in multiple-choice VQA make benchmarking unreliable

The paper demonstrates that multiple‑choice visual question answering (MC‑VQA) benchmarks are unreliable because model performance is highly sensitive to semantically neutral prompt formatting choices—such as option ID sets, delimiters, and separators—despite protocols that mitigate option‑order effects. Across seven multimodal large language models and five datasets, the authors observed frequent rank reversals when systematically varying 48 equivalent prompt formats, attributing the instability to tokenizer‑induced token fusion or removal and to how option ID sets influence attention patterns. Consequently, MC‑VQA rankings correlate weakly with open‑ended evaluation, revealing that MC‑VQA reflects option‑selection dynamics as well as multimodal reasoning.

By Fabio Rosenthal, Sebastian Schmidt, Thorsten Graf, Thorsten Bagdonat, Stephan G\"unnemann, Leo Schwinn
arXiv Machine Learning
Sep 10

SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

arXiv:2609.05533v1 Announce Type: cross Abstract: Long-horizon manipulation is partially observable: the information needed to choose the next action may appear only in observations from minutes earl...

By Cheng Yin, Wang Xu, Junpeng Yang, Sikyuen Tam, Hanyu Liu, Yuan Yao, Xiangrui Zeng, Junbo Cui, Yequan Wang, Zhouping Yin, Yankai Lin
arXiv AI
Sep 10

MemForest: Efficient Agent Memory Management via EventTree Partitioning and Progressive Merging

arXiv:2609.08273v1 Announce Type: new Abstract: Agent memory systems have demonstrated significant potential in long-term dialogue, personalized assistants, and video understanding. However, continuo...

By Junxi Wang, Te Sun, Jiayi Zhu, Chen Zhang, Siyuan Li, Xuyang Liu, Zichen Wen, Xiaobing Tu, Jinkui Ren, Xiantao Zhang, Ziqi Yuan, Linfeng Zhang