← Back to all news

Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,624 stories · RSS feed

arXiv Computer Vision
Aug 25

Toward a Foundation Plug-and-Play Prior for Computed Tomography Reconstruction via a Multimodal Diffusion Model

The paper proposes a single diffusion model trained across multiple imaging domains to act as a reusable prior for computed tomography (CT) reconstruction. By keeping the model frozen, it is applied to three distinct datasets—flaw analysis in additively manufactured metal parts using cone‑beam X‑ray CT, and concrete microstructure imaging with parallel‑beam neutron CT—each differing in modality, geometry, material, and degradation. In all cases, the method outperforms analytic reconstructions, demonstrating its potential as a foundation plug‑and‑play prior for heterogeneous CT problems.

By Haley Duba-Sullivan, Patxi Fernandez-Zelaia, Obaidullah Rahman, Amirkoushyar Ziabari
diffusionefficiencymultimodal
More like this →
arXiv Computer Vision
Aug 25

Region-Weighted Losses and Model Fusion for Cross-Modal PET Attenuation Correction

The paper presents a method for the BIC-MAC challenge, aiming to generate pseudo‑CT images from NAC‑PET, DIXON MRI, and a topogram, and to evaluate both the pseudo‑CT and the resulting attenuation‑corrected PET. The authors improve upon a 3D U‑Net baseline by focusing on loss design: they compute an L1 error in the Carney attenuation‑coefficient space, weighted by anatomical region, and incorporate DIXON MRI as additional input only after this loss was applied. Finally, they fuse two independently trained models via a fixed convex combination, achieving better performance than either model alone and topping the public validation leaderboard.

By Khoa Tuan Nguyen, Joris Vankerschaver, Wesley De Neve
multimodalbenchmarks
More like this →
arXiv AI
Aug 25

ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation

ATP‑Bench proposes a new benchmark for evaluating agentic tool planning in multimodal large language models (MLLMs) that generate interleaved text-and-image responses. The benchmark contains 7,702 QA pairs, including 1,592 visual‑question‑answer pairs, across eight categories and 25 visual‑critical intents, all verified by humans. A Multi‑Agent MLLM‑as‑a‑Judge (MAM) system is introduced to assess tool‑call precision, missed opportunities, and overall response quality without relying on ground‑truth references.

By Yinuo Liu, Zi Qian, Heng Zhou, Jiahao Zhang, Yajie Zhang, Zhihang Li, Mengyu Zhou, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang
llmsagentsdiffusionmultimodalbenchmarks
More like this →
arXiv AI
Aug 25

GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis

arXiv:2608.22847v1 Announce Type: new Abstract: Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottl...

By Long Zhang, Yuhan Chen, Chaoran Zhang, Wanxia Cao, Kun Huang, Pengzhi Gao, Wei Liu, Jian Luan, Chenliang Li, Lixin Zou
llmsagentsreinforcement-learningmultimodalbenchmarks
More like this →
arXiv Computer Vision
Aug 25

SketchFlow: Zero-Shot Vector Sketch Generation via GMM Prior Flow in CLIP Latent Space

SketchFlow is a new generative framework for creating high‑quality vector sketches from text prompts. It uses a Gaussian Mixture Model prior in the CLIP latent space and an Optimal Transport Conditional Flow Matching model to map this prior to sketch features, which are then decoded by a Hybrid Diffusion Decoder combining 1D U‑Net and Transformer architectures. The approach achieves superior visual quality and human‑like drawing styles, and supports zero‑shot synthesis for unseen concepts and smooth semantic interpolation.

By Jin Zhou, Hongliang Yang, Pengfei Xu, Hui Huang
llmsragdiffusionmultimodalsafety
More like this →
arXiv AI
Aug 25

Budget-Constrained Embodied Perception: Four Resource Walls and a Pre-Registered Evaluation of Access-Structured Perception on Open Models at less than 31B

arXiv:2608.22975v1 Announce Type: new Abstract: Embodied multimodal agents must answer from growing observation streams under a fixed per-decision token budget. We formalize this constraint through f...

By Defu Lin, Wenhui Chen, Ziyao Lin, Jianlin Chen, Peiji Long, Chi Man Vong
agentsroboticsmultimodalbenchmarks
More like this →
arXiv Computer Vision
Aug 25

TONAV: Task-Oriented Navigation and Action-Velocity Chunk Learning for Articulated Object Quadrupedal Mobile Manipulation

arXiv:2608.22296v1 Announce Type: cross Abstract: Quadruped mobile manipulation requires two tightly coupled capabilities: reaching manipulation-ready configurations and maintaining stable contact th...

By Haoran Lin, Mingyu Yang, Pengfei Qi, Kehan Chen, Qiang Diao, Liangji Zeng, Wenrui Chen, Yaonan Wang, Kailun Yang
roboticsmultimodal
More like this →
arXiv Computer Vision
Aug 25

Predicting Multiple Clinical Outcomes Related to Functional Recovery and Social Isolation Among Older Adults After Lower-Limb Fracture or Hip Replacement

arXiv:2608.23531v1 Announce Type: new Abstract: Older adults recovering after lower-limb fracture or hip replacement may experience complex recovery trajectories. Most of the time, these clinical asp...

By Santosh Ray, Pratik K. Mishra, Ali Abedi, Charlene H. Chu, Amir Ahmad, Shehroz S. Khan
multimodal
More like this →