Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Computation and Language

Oct 15

When More Documents Hurt RAG: Measuring and Reducing Vector Search Dilution with Domain-Scoped Retrieval

arXiv:2606.11350v2 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) degrades when applied to large, heterogeneous document collections. In RAG systems, dense similarity becomes l...

By Nabaraj Subedi, Ahmed Abdelaty, Shivanand Venkanna Sheshappanavar

llms rag agents benchmarks

arXiv Computation and Language

Oct 15

DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents

arXiv:2606.17029v2 Announce Type: replace Abstract: Deep research agents synthesize long-form reports by searching and reasoning over retrieved evidence. Reinforcement learning with rubric-based rewa...

By Minghang Zhu, Chuyang Wei, Junhao Xu, Yilin Cheng, Zhumin Chen, Jiyan He

llms agents reinforcement-learning benchmarks

arXiv Computation and Language

Oct 15

Confidently Wrong: Detecting Hallucinations in Financial Question Answering from LLM Internal States

arXiv:2607.11414v2 Announce Type: replace Abstract: Large language models (LLMs) in financial applications fail most consequentially when they are confidently wrong. Hedged, uncertain answers invite...

By Richard Zhe Wang

llms nlp benchmarks safety

arXiv Computation and Language

Oct 15

Reward Design for Physical Reasoning in Vision-Language Models

arXiv:2604.13993v2 Announce Type: replace-cross Abstract: Physical reasoning over images requires integrating perception, domain knowledge, and multi-step symbolic inference, yet Vision-Language Mode...

By Derek Lilienthal, Manisha Mukherjee, Sameera Horawalavithana

llms reinforcement-learning multimodal benchmarks

arXiv Computation and Language

Oct 15

Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval

arXiv:2604.18360v4 Announce Type: replace-cross Abstract: Audio-text retrieval systems based on Contrastive Language-Audio Pretraining (CLAP) achieve strong performance on traditional benchmarks; how...

By HaeJun Yoo, Yongseop Shin, Insung Lee, Myoung-Wan Koo, Du-Seong Chang

llms multimodal benchmarks

arXiv Computation and Language

Oct 15

AVA-Encoder: Towards Agent-Native Video Representation Learning

arXiv:2608.12313v3 Announce Type: replace-cross Abstract: Agentic video creation systems remain far from cinematic-grade quality: their underlying models struggle to plan how story, characters, visua...

By Chuyue Li, Jinpeng Yu, Haozhe Wang, Tian Xueyun, Zhijing Zhang, Bingnan Li, Shuqi Gu, Kan Ren, Jiaming Liu, Ruihua Huang

agents multimodal benchmarks

arXiv Computer Vision

Oct 15

Action Jets: Learning the Local Shape of the Action-conditioned Data Manifold

arXiv:2610.19007v1 Announce Type: new Abstract: World models are a powerful means of reasoning and planning, but many of these action-conditioned predictive models are high dimensional and computatio...

By Anton Arapin, Chahyon Ku, Bernadette Bucher

robotics benchmarks

arXiv Computer Vision

Oct 15

The Gap Is in the Geometry: A Convergent Meta-Analysis of Multi-Facet Aerial and Remote Sensing Video Understanding

arXiv:2610.19102v1 Announce Type: new Abstract: Aerial and remote sensing video is increasingly evaluated like web video, by posing multiple-choice questions to multimodal large language models (MLLM...

By Georgios Angelidis, Aggelos Psiris, Maria Vakalopoulou, Georgios Th. Papadopoulos, Giorgos Sfikas, Iraklis Varlamis

llms multimodal benchmarks

arXiv Computer Vision

Oct 15

RDCompGS: Rate-Distortion Optimization for Compression of 3D Gaussian Splatting with Image Codecs

arXiv:2610.19538v1 Announce Type: new Abstract: 3D Gaussian Splatting (3DGS) enables high-quality, real-time novel view synthesis but requires significant storage and transmission resources. Mapping...

By Huiping Ren, Shuai Wan, Luqian Bai, Zhang Chen, Fuzheng Yang

benchmarks

arXiv Computer Vision

Oct 15

PhyMetric: Diagnosing Physical Plausibility in Text-to-Video Generation via Scene-Level QA

arXiv:2610.19823v1 Announce Type: new Abstract: Text-to-video (T2V) generation models can produce visually realistic videos that nevertheless violate the physical consequences implied by the prompt a...

By Siwei Meng, Xuerou Cai, Rui Hu, Ping Liu

llms diffusion nlp multimodal benchmarks safety

arXiv Computer Vision

Oct 15

Glance Forcing: Making Bidirectional Diffusion Autoregressive at a Glance

arXiv:2610.20220v1 Announce Type: new Abstract: Autoregressive generation underlies real-time, causal video synthesis and is the backbone of video world models. Today's dominant approach distills a b...

By Zhuobai Dong, Junchao Yi, Hu Jian Guo, Linjie Li, Rui Zhao, Alex Jinpeng Wang

diffusion efficiency benchmarks

arXiv Computer Vision

Oct 15

VisualWorldBench: A Fine-Grained Multi-Task Benchmark for Evaluating Visual World Knowledge in MLLMs

arXiv:2610.20083v1 Announce Type: new Abstract: Visual world knowledge is a core capability of multimodal large language models (MLLMs), underpinning knowledge-intensive visual question answering and...

By Chen Chen, Zheming Liang, Jiaqi Wang

llms nlp fine-tuning multimodal benchmarks

arXiv Computer Vision

Oct 15

PAWA: Panoramic Action-conditioned World Modeling for Multiple Agents

arXiv:2610.20240v1 Announce Type: new Abstract: A multiplayer world model must predict how the independent actions of several players change a shared environment, including changes outside a player's...

By Siqi Yang, Yimu Wang, Ziqi Cai, Chengxuan Zhu, Shuchen Weng, Erwin Wu, Zhaofei Yu, Kaipeng Zhang, Boxin Shi

rag agents benchmarks

arXiv Computer Vision

Oct 15

Beyond Static Cues: Unified Dynamic Detection of MLLM Backdoors via Dual-Flow Trajectory Analysis

arXiv:2610.20319v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) are increasingly deployed in safety-critical domains, yet their susceptibility to backdoor attacks remains a f...

By Yinzhen Huang, Jiahe Xie, Yitong Sun, Ruochen Zhang, Xingxing Wei

llms multimodal benchmarks safety

arXiv Computer Vision

Oct 15

InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Views

arXiv:2610.20341v1 Announce Type: new Abstract: Feed-forward Gaussian reconstruction builds 3D scenes directly from input images for novel view rendering, but learning Gaussian attributes from predic...

By Yongzhen Hu, Hao Yu, Jiawei Wang, Xianming Zeng, Jianyun Xu, Xingliang Liu, Ruizhen Hu, Xiaowei Zhou, Hujun Bao, Sida Peng

benchmarks

arXiv Computer Vision

Oct 15

OLiVES: An Outdoor Low-Light Video Benchmark for Enhancement and Segmentation

arXiv:2610.20501v1 Announce Type: new Abstract: Low-light video processing remains an underexplored problem in computer vision, primarily due to the scarcity of high-quality ground-truth data. Existi...

By Yini Li, Joanne Lin, David Blackham, David Bull, Nantheera Anantrasirichai

computer-vision benchmarks

arXiv Computer Vision

Oct 15

Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generation

arXiv:2610.20571v1 Announce Type: new Abstract: Long-form agentic video generation must turn a screenplay into shots that remain spatially and temporally consistent across nonconsecutive requests. Pa...

By Weijie Wang, Haomin Zhang, Ye Li, Akide Liu, Zeyu Zhang, Haoyu Zhao, Yefei He, Donny Y. Chen, Feng Chen, Bohan Zhuang

agents diffusion benchmarks

arXiv Computer Vision

Oct 15

ELVIS: Enhance Low-Light for Video Instance Segmentation in the Dark

arXiv:2512.01495v4 Announce Type: replace Abstract: Video instance segmentation (VIS) for low-light content remains highly challenging for both humans and machines alike, due to noise, blur and other...

By Joanne Lin, Ruirui Lin, Yini Li, David Bull, Nantheera Anantrasirichai

computer-vision fine-tuning benchmarks

arXiv Computer Vision

Oct 15

CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregation

arXiv:2601.08010v4 Announce Type: replace Abstract: Vision-language models achieve strong performance across a wide range of multimodal understanding and reasoning tasks, yet their multi-step reasoni...

By Chaoyu Li, Fei Tao, Pooyan Fazli

llms reinforcement-learning multimodal benchmarks

arXiv Computer Vision

Oct 15

GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos

arXiv:2603.14426v2 Announce Type: replace Abstract: Existing text-to-video retrieval benchmarks often permit matching from static appearance, making it difficult to determine whether models capture s...

By Minghan Li, Tongna Chen, Tianrui Lv, Yishuai Zhang, Suchao An, Guodong Zhou

diffusion robotics benchmarks

Get the daily briefing