← Back to all news
arXiv Computation and Language By Alexander Martin, William Walden, Reno Kriz, Dengjia Zhang, Kate Sanders, Eugene Yang, Chihsheng Jin, Benjamin Van Durme

Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation

Read the original on arXiv Computation and Language →

MiRAGE is a new evaluation framework for retrieval‑augmented generation (RAG) that handles multimodal sources such as audiovisual media. It uses a claim‑centric approach with two metrics: InfoF1, which measures factuality and information coverage, and CiteF1, which measures citation support and completeness. Human evaluation shows MiRAGE aligns well with extrinsic quality judgments, and an automatic implementation outperforms three text‑centric RAG metrics (ALCE, ARGUE, RAGAS) on text while uniquely generalizing to multimodal inputs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Sep 18

Less Is More: Graph-free Multimodal RAG via Multi-signal Late Fusion

The paper introduces TrioRAG, a graph-free multimodal retrieval-augmented generation framework that combines evidence from the question, an anchor image, and a VLM-enhanced query via late fusion. It also presents AutoQA, a benchmark featuring noisy web-sourced images that require reasoning across manuals. TrioRAG outperforms graph-based systems on three benchmarks while cutting costs and speeding up inference by 1.6–2.3×.

By Tithi Rakshit, Hongkuan Zhou, Lavdim Halilaj, Yuqicheng Zhu
ragnlpmultimodalbenchmarks
More like this →
arXiv AI
Jun 26

MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation

arXiv:2606. 26458v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) over knowledge graphs has emerged as a promising approach for grounding large language models, yet existing benchmarks largely overlook the challenges of retrieval in multimodal knowledge graph RAG (MKG-RAG).

By Xiaochen Wang, Bao Hoang, Han Liu, Ting Wang, Fenglong Ma
llmsragmultimodalbenchmarks
More like this →
arXiv AI
Jul 21

A Survey on Knowledge-Oriented Retrieval-Augmented Generation

arXiv:2503. 10677v3 Announce Type: replace-cross Abstract: Retrieval-Augmented Generation (RAG) has gained significant attention in recent years for its potential to enhance natural language understanding and generation by combining large-scale retrieval systems with generative models.

By Mingyue Cheng, Yucong Luo, Jie Ouyang, Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, Daoyu Wang, Enhong Chen
ragnlpmultimodalbenchmarkssafety
More like this →
arXiv AI
Jun 16

VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA

arXiv:2606. 16092v1 Announce Type: cross Abstract: Real-world documents combine text with tables, charts, photographs, and diagrams arranged in diverse layouts, yet existing research on multimodal large language models (MLLMs) for document QA predominantly produces text-only responses, underutilizing these visual elements.

By Young Rok Jang, Hyesoo Kong, Kyunghwan An, Jae Sub Huh, Gyeonghun Kim, Stanley Jungkyu Choi
llmsfine-tuningmultimodal
More like this →
arXiv AI
2d ago

MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

arXiv:2505.17613v2 Announce Type: replace Abstract: Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation...

By Jihan Yao, Yushi Hu, Wenyuan Wang, Bin Han, Shangbin Feng, Guang Yang, Yujie Yi, Bingbing Wen, Ranjay Krishna, Lucy Lu Wang, Yulia Tsvetkov, Noah A. Smith, Banghua Zhu
llmsdiffusionmultimodalbenchmarks
More like this →