← Back to all news

Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI
Oct 2

Two Clocks in Diffusion MLLMs: When Answers Stabilize Before Rationales Unfold

The paper investigates how answer candidates in masked diffusion multimodal large language models (MLLMs) can stabilize before their rationales are fully generated. It distinguishes between retrospective stabilization of the logged candidate and token commitment, and evaluates these two temporal markers across three visual question‑answering benchmarks. The study finds that a large portion of the rationale canvas remains unwritten at stabilization, that reducing block length dramatically lowers this fraction, and that direct prompting instructions can significantly alter accuracy depending on the model and dataset. The authors also use matched‑canvas image ablations to separate visual sensitivity from answer stabilization, concluding that coverage rather than conditional accuracy drives most prompting differences.

By Keuntae Kim, Yong Suk Choi
llmsdiffusionbenchmarks
More like this →
arXiv AI
Oct 2

Divide-and-Remember: Recursive Action-Relevant Memory for Long-Horizon VLA Policies

The paper introduces Divide-and-Remember (D&R), a recursive memory method for vision-language-action (VLA) policies that optimises memory by maximizing the conditional mutual information between actions and memory given observations. D&R recursively divides the full history into top‑K selections over 2K tokens, using a shared lightweight selector across all recursion blocks to handle unbounded histories efficiently. Evaluated on the RoboMME benchmark of 16 long‑horizon manipulation tasks, D&R achieves state‑of‑the‑art success rates with consistent gains across all suites while using only 64 tokens, and similar improvements are observed in real‑robot experiments.

By Xuehui Yu, Eason Yu, Meiyi Wang, Haozhe Du, Stefano V. Albrecht, Harold Soh
roboticsmultimodalbenchmarks
More like this →
arXiv AI
Oct 2

JoinGR: Learning to Traverse Join Graphs for Table Retrieval

JoinGR is a join‑aware table retrieval method that treats a database’s join graph as the retrieval space. It represents columns as graph nodes and both intra‑table and foreign‑key relationships as typed edges, then selects anchor tables, traverses join edges with a query‑conditioned scorer, and aggregates edge deposits into table scores. Experiments on BIRD, Spider, and BEAVER show that JoinGR matches or surpasses strong retrieval baselines, especially improving recall on multi‑hop table requirements, and its learned scorer transfers across benchmarks, indicating reusable join‑graph traversal behavior.

By Sandipan De, Abhijit Chakraborty, Sambaran Bandyopadhyay, Vivek Gupta
ragbenchmarksdata-engineering
More like this →
arXiv AI
Oct 2

CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessment

CineMR is a vision‑language model that integrates cardiac image‑analysis tools to perform quantitative assessment of cine cardiac MRI. It uses supervised fine‑tuning and Group Relative Policy Optimization to learn reliable tool invocation, achieving significantly higher accuracy on a multi‑cohort benchmark than existing medical VLMs. The model demonstrates that tool‑augmented reasoning improves ventricular measurement accuracy by up to 23.7% and is essential for robust quantitative CMR interpretation.

By Kunyang Li, Hai Nguyen, Joshua Lowe, Chenguang Zhao, Peace C. Madueme, Mehdi Hedjazi Moghari, Mubarak Shah, Pegah Khosravi, Yuzhang Zhang
llmsagentscomputer-visionnlpfine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Oct 2

A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models

The paper introduces a benchmark for evaluating large language models on trustworthy analysis of earnings call transcripts. It proposes a numeric evidence evaluation method that assesses groundedness without expert annotation, and presents an automated pipeline that builds the ECTs-100 dataset from the top 100 S&P 500 constituents. The study also explores the failure mode of conscious incompetence, where models must recognize insufficient evidence and avoid hallucinations, finding that while groundedness is strong, correctness remains a challenge.

By Yingzhu Zhao, Vlad Pandelea, Han Yuan, Bo Hu, Wuqiong Luo, Li Zhang, Zheng Ma
llmsbenchmarkssafety
More like this →
arXiv AI
Oct 2

Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs

The paper introduces Temporal Activation Injection (TAI), a post‑processing technique that reinforces temporal representations in Video Large Language Models (VideoLLMs) at inference time. By measuring the temporal divergence vector τ_l— the layer‑wise difference caused by reversing video frame order— the authors find that temporal information peaks at intermediate layers and fades toward the output. TAI injects the peak divergence back into later layers, improving temporal reasoning on three VideoLLMs across four benchmarks without affecting non‑temporal performance.

By Youngwoo Shin, Yusung Ro, Minseo Kim, Junmo Kim
llmsbenchmarks
More like this →
arXiv AI
Oct 2

Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding

Cog-VADU is a training‑free framework that transforms video anomaly detection into a sequential cognitive reasoning task. It uses Chain‑of‑Anomaly Detection Thought Prompting (CoADTP) to create a recurrent reasoning chain across video segments, preserving temporal memory and distinguishing complex anomalies from high‑motion normal activities. A cross‑modal re‑ranking stage aligns textual rationales with visual embeddings to enforce semantic consistency and temporal coherence, yielding competitive zero‑shot performance on multiple VAD benchmarks.

By Mohd Ubaid Wani, Sara Atito, Josef Kittler, Muhammad Awais
llmsragmultimodalbenchmarks
More like this →
arXiv AI
Oct 2

MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

arXiv:2505.17613v2 Announce Type: replace Abstract: Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation...

By Jihan Yao, Yushi Hu, Wenyuan Wang, Bin Han, Shangbin Feng, Guang Yang, Yujie Yi, Bingbing Wen, Ranjay Krishna, Lucy Lu Wang, Yulia Tsvetkov, Noah A. Smith, Banghua Zhu
llmsdiffusionmultimodalbenchmarks
More like this →