Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computation and Language

Sep 22

You Frame It: How Conceptual Representations Shape LLM Detection and Reasoning about Antisemitism

arXiv:2607.04945v2 Announce Type: replace Abstract: LLMs enable the integration of external conceptual resources at inference time, creating new opportunities for detecting ideologically and historic...

By Katharina Soemer, Helena Mihaljevi\'c

llms benchmarks

arXiv Computation and Language

Sep 22

LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization

arXiv:2608.18082v2 Announce Type: replace Abstract: Although context windows have expanded significantly in recent years, hallucinations in long-context summarization remain a challenge. Long novels...

By Ruizhi Zhang, Jinwei Chen, Xiangju Lu, He Yan, Mo Yu, Junmin Zhu, Wei Zhang

nlp benchmarks safety

arXiv Computation and Language

Sep 22

SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models

arXiv:2508.03351v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across diverse language tasks, motivating their extension to vision-la...

By Yufei Xue, Yushi Huang, Lunjie Zhu, Jiawei Shao, Jun Zhang

llms efficiency multimodal benchmarks

arXiv Computation and Language

Sep 22

A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents

arXiv:2601.12538v2 Announce Type: replace-cross Abstract: Reasoning is a fundamental cognitive process underlying inference, problem-solving, and decision-making. While large language models (LLMs) d...

By Tianxin Wei, Ting-Wei Li, Zhining Liu, Xuying Ning, Ze Yang, Jiaru Zou, Zhichen Zeng, Ruizhong Qiu, Xiao Lin, Dongqi Fu, Zihao Li, Mengting Ai, Duo Zhou, Wenxuan Bao, Yunzhe Li, Gaotang Li, Cheng Qian, Yu Wang, Xiangru Tang, Yin Xiao, Liri Fang, Hui Liu, Xianfeng Tang, Yuji Zhang, Chi Wang, Jiaxuan You, Heng Ji, Hanghang Tong, Jingrui He

llms agents reinforcement-learning robotics fine-tuning benchmarks

arXiv Computer Vision

Sep 22

Did You Steal My Shot? Pioneering Camera Motion Plagiarism Detection in Generative Videos

arXiv:2609.22267v1 Announce Type: new Abstract: Camera motion often reflects directorial intent and requires professional equipment, making it a high value form of intellectual property. However, gen...

By Chengguo Zhang, Ping Ping

computer-vision benchmarks

arXiv Computer Vision

Sep 22

Validating, Not Sampling: Region-Level Robustness of Vision-Language and Vision-Language-Action Models

arXiv:2609.22293v1 Announce Type: new Abstract: Vision-language models (VLMs) and vision-language-action models (VLAs) are increasingly deployed in real-world applications. There, a small perturbatio...

By Bogdan Aron, Christopher Brix, Benedikt Br\"uckner, Yanghao Zhang, Panagiotis Kouvaros, Alessio Lomuscio

llms multimodal benchmarks

arXiv Computer Vision

Sep 22

Hierarchical Aggregation of Semantic Uncertainty in 3D Scene Graphs

arXiv:2609.22351v1 Announce Type: new Abstract: Open-vocabulary 3D Scene Graphs (3DSGs) ground each object node in a vision-language embedding, yet they record every entry as equally certain, so a ro...

By Carlos Cueto Zumaya, Iacopo Catalano, Wallace Moreira Bessa, Julio A. Placed

llms rag robotics multimodal benchmarks

arXiv Computer Vision

Sep 22

GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents

arXiv:2609.22308v1 Announce Type: new Abstract: Coding-agent benchmarks usually evaluate implementation after the target behavior has been specified in text, code, or demonstrations. Existing researc...

By Boyu Qiao, Zixin Tang, Xiaoshuai Hao, Wenbo Li

llms agents multimodal benchmarks

arXiv Computer Vision

Sep 22

AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval

arXiv:2609.22562v1 Announce Type: new Abstract: Multi-vector visual document retrieval (VDR) models such as ColPali and ColNomic achieve strong accuracy by representing each document with hundreds to...

By Jianxin You, Kun Ni

rag efficiency benchmarks

arXiv Computer Vision

Sep 22

Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act

arXiv:2609.22588v1 Announce Type: new Abstract: Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they h...

By Yuyang Dai, Bofei Huang, Hongbo Zhang, Haoran Xie

llms nlp multimodal benchmarks

arXiv Computer Vision

Sep 22

X-Beat: An Explainable Framework for ECG Image Classification

arXiv:2609.22631v1 Announce Type: new Abstract: Accurate automated interpretation of electrocardio- grams (ECGs) is essential for early detection of cardiac condi- tions such as myocardial infarction...

By Mohammad Sadman Tahsin, Haitham Y. Adarbah, Afzel Noore

computer-vision fine-tuning benchmarks

arXiv Computer Vision

Sep 22

Vision2CAD: A Visual Agent Harness for Explicit Geometry Referencing and Localization in Parametric CAD Modeling

arXiv:2609.22688v1 Announce Type: new Abstract: Generating parametric CAD models requires accurate geometry and stable feature dependencies. Existing methods face challenges in selecting geometric re...

By Xi Cheng, Chenxi Zhai, Hang Cheng, Mingyu Fan, Pingfa Feng, Long Zeng

llms agents multimodal benchmarks

arXiv Computer Vision

Sep 22

PixelART: Image-to-Layer Decomposition without Latents or Text-to-Image Pretraining

arXiv:2609.22789v1 Announce Type: new Abstract: Image-to-layer decomposition converts a flattened image into editable RGBA layers, enabling element-level editing in design workflows. Existing diffusi...

By Zelin Jia, Zhao Zhang, Zhicong Tang, Yuhui Yuan, Shixia Liu

llms diffusion multimodal benchmarks

arXiv Computer Vision

Sep 22

SatOV: Restoring Spatial Priors for Training-Free Open-Vocabulary Segmentation in Remote Sensing Imagery

arXiv:2609.22834v1 Announce Type: new Abstract: Open-vocabulary semantic segmentation (OVS) of remote sensing imagery is a challenging pixel-level task requiring strong generalization and adaptation...

By Changhao Zhao, Linglin Zeng, Hai Liu

computer-vision multimodal benchmarks

arXiv Computer Vision

Sep 22

Combining Foundation Model Confidence and Monocular Depth for Training-Free Out-of-Distribution Segmentation

arXiv:2609.22896v1 Announce Type: new Abstract: Autonomous vehicles operating in open-world scenarios are inevitably confronted with previously unknown objects, such as exotic animals or loose cargo....

By Serin Varghese, Fabian H\"uger, Kira Maag

agents computer-vision fine-tuning benchmarks

arXiv Computer Vision

Sep 22

D3GS: Depth, DINO, and RGB Diffusion Co-Guided 3D Gaussian Splatting for Sparse-View Reconstruction

arXiv:2609.22941v1 Announce Type: new Abstract: Novel view synthesis from sparse inputs remains challenging for 3D Gaussian Splatting (3DGS) due to ambiguous geometry, cross-view inconsistency, and m...

By Yunqi Gao, Zhanfeng Liao, Hanzhang Tu, Zhaoqi Su, Guoqing Zheng, Songtao Wang, Hongwen Zhang, Zhou Xue, Leyuan Liu, Yebin Liu

llms diffusion benchmarks

arXiv Computer Vision

Sep 22

CrowdCue: Specialist-Cue Conditioning for Vision-Language Crowd Counting

arXiv:2609.23012v1 Announce Type: new Abstract: Generative vision-language models (VLMs) offer a counting paradigm in which one model produces both a count and a natural-language account of the scene...

By Moshiur Farazi, Bekir Ciftler, Abdulhalim Dandoush, Reda Bendraou

llms fine-tuning multimodal benchmarks safety

arXiv Computer Vision

Sep 22

MM-ContextFold: Context Folding for Multimodal Agentic Retrieval

arXiv:2609.23121v1 Announce Type: new Abstract: Multimodal Agentic Retrieval (MAR) requires agents to solve complex information-seeking tasks by iteratively invoking external tools. Typical framework...

By Yang Tian, Fan Liu, Jingyuan Zhang, Zhenyang Li, Yupeng Hu, Liqiang Nie

agents multimodal benchmarks

arXiv Computer Vision

Sep 22

LiteTex-GS: Fast and Lightweight Texturing for Gaussian Splatting

arXiv:2609.23380v1 Announce Type: new Abstract: Gaussian Splatting has enabled real-time novel view synthesis, but its tightly coupled geometry and appearance representation often require a large num...

By Zhiwei Li, Yijia Guo, Yishi Lu, Liwen Hu, Hong Rao, Shengbo Chen, Lei Ma

efficiency benchmarks

arXiv Computer Vision

Sep 22

Accurate Motion Estimation with B\'ezier Control Point for Efficient Frame Interpolation

arXiv:2609.23408v1 Announce Type: new Abstract: In frame interpolation tasks, motion ambiguity in the training set causes models to generate blurry intermediate frames. Moreover, the assumption of un...

By Shuhao Han, Chenyang Wu, Chun-Le Guo, Zheng-Peng Duan, Zhen Li, Ming-Ming Cheng, Chongyi Li

computer-vision fine-tuning benchmarks

Get the daily briefing