Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computation and Language

Oct 27

CRIM: Composable and Reusable Item Memory for Efficient Personalized Retrieval

arXiv:2610.35323v1 Announce Type: cross Abstract: Large language models incur substantial training and inference costs in personalized search and recommendation, as growing interaction histories leng...

By Haolin Li, Haipeng Zhang, Mang Li, Zhaohui Xia, Xun Zhu, Miao Li, Yu Zhang, Ji Wu

llms benchmarks

arXiv Computation and Language

Oct 27

Reinforcement Learning Done Right for Explicit Confidence Targets

arXiv:2610.35059v1 Announce Type: cross Abstract: Large language models (LLMs) can produce plausible but incorrect answers when they are uncertain, which can mislead users and undermine model reliabi...

By Zefeng He, Yichen He, Yuan Lin, Yang Yang, Yu Cheng

llms reinforcement-learning benchmarks

arXiv Computation and Language

Oct 27

KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues

arXiv:2509.21856v2 Announce Type: replace Abstract: Multi-Turn Long-Form Question Answering (MT-LFQA) is a key application paradigm of Large Language Models (LLMs) in knowledge-intensive domains. How...

By Junhao Chen, Yu Huang, Siyuan Li, Rui Yao, Hanqian Li, Hanyu Zhang, Jungang Li, Jian Chen, Bowen Wang, Xuming Hu

llms rag nlp fine-tuning benchmarks

arXiv Computation and Language

Oct 27

The Lexical Imprint of Pre-training Data

arXiv:2602.10657v2 Announce Type: replace Abstract: How much of zero-shot benchmark accuracy is captured by word-level alignment between a pre-training corpus and the benchmark target? We study this...

By Jeonghoon Kim, Woojin Chung, Woomin Song, Cheonbok Park, Nako Sung, Jinwoo Shin

nlp efficiency benchmarks safety

arXiv Computation and Language

Oct 27

Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification

arXiv:2605.00326v2 Announce Type: replace Abstract: Safety pipelines often block content by thresholding a model's first-token unsafe score. We show that a single prompt's score is not a reliable pro...

By Charles Weng, Dingwen Li, Alexander Martin

llms multimodal benchmarks safety

arXiv Computation and Language

Oct 27

Detecting Differences Is Not Understanding Structure: Large Language Models Fail at Graph Isomorphism

arXiv:2606.09484v2 Announce Type: replace Abstract: Large language models (LLMs) have shown impressive performance on diverse reasoning tasks, yet their capacity for structural reasoning in graphs re...

By Kumar Thushalika, Sukumar Kishanthan, Asela Hevapathige

llms benchmarks

arXiv Computation and Language

Oct 27

PriFT: Guiding Supervised Fine-Tuning with Pretrained Priors

arXiv:2606.09396v2 Announce Type: replace Abstract: Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation and often serves as the initialization stage for reinforcement...

By Ke Wang, Shuangqi Li, Mathieu Salzmann, Pascal Frossard

reinforcement-learning fine-tuning benchmarks

arXiv Computation and Language

Oct 27

P2N: Reusing Deep Representations for Greater Effective Depth

arXiv:2610.25076v2 Announce Type: replace Abstract: Mathematical problem solving and autonomous agents are increasingly important applications of language models, demanding stronger reasoning capabil...

By Yiqin Hao, Boyi Zeng, Zitong Wang, He Li, Shixiang Song, Feichen Song, Yifan Liu, Ran Tao, Bryan Dai, Zhouhan Lin

llms agents benchmarks

arXiv Computation and Language

Oct 27

Truncated Reasoning Under a Budget: Evaluating System-1 Models as Decision-Makers

arXiv:2610.25768v2 Announce Type: replace Abstract: Large language models (LLMs) often generate more reasoning tokens than needed. Limiting the reasoning budget reduces cost, but forcing an answer fr...

By Zhongyin Wang, Zhaoqian Xue, Junting Mu, Haozheng Liu, Ye Yuan, Haolun Wu, Lingyao Li

llms benchmarks

arXiv Computation and Language

Oct 27

Can Multi-modal (reasoning) LLMs detect document manipulation?

arXiv:2508.11021v2 Announce Type: replace-cross Abstract: Document fraud poses a significant threat to industries reliant on secure and verifiable documentation, necessitating robust detection mechan...

By Zisheng Liang, Kidus Zewde, Rudra Pratap Singh, Disha Patil, Zexi Chen, Jiayu Xue, Yao Yao, Yifei Chen, Qinzhe Liu, Simiao Ren

llms robotics fine-tuning multimodal benchmarks

arXiv Computation and Language

Oct 27

Automotive-ENV: Benchmarking Multimodal Models in Automotive Cockpit Environments

arXiv:2509.21143v3 Announce Type: replace-cross Abstract: Multimodal models have demonstrated strong capabilities in web, desktop, and mobile GUI environments, but their performance in automotive coc...

By Junfeng Yan, Biao Wu, Meng Fang, Ling Chen

llms multimodal benchmarks safety

arXiv Computation and Language

Oct 27

SATFormer: Learning When to Use Early Value Residuals in Transformers

arXiv:2605.03953v3 Announce Type: replace-cross Abstract: Recent Transformer architectures expose later layers to representations computed in earlier layers, motivated by the observation that low-lev...

By Skye Gunasekaran, T\'ea Wright, Rui-Jie Zhu, Jason Eshraghian

llms efficiency benchmarks

arXiv Computer Vision

Oct 27

EKT-Router: Expert-Evidence Alignment for Embodied Vision-Language Models

arXiv:2610.33044v1 Announce Type: new Abstract: Embodied vision-language models must answer questions from partial, evolving observations. Their success depends on both the reasoning capability selec...

By Weilin Ruan, Yingjie Xu, Yangkai Wei, Bohan Zhou, Shuaibo Li, Harold Haodong Chen, Kaiwen Zhou, Yinchuan Li, Ying-Cong Chen, James Cheng

llms robotics multimodal benchmarks safety

arXiv Computer Vision

Oct 27

OvisOCR3 Technical Report

arXiv:2610.33066v1 Announce Type: new Abstract: We introduce OvisOCR3, a 0.8B model for a broad range of OCR-related tasks. OvisOCR3 unifies document parsing, general OCR, key information extraction...

By Yu Xia, Yinglun Li, Yuhui Chen, Jinlong Huang, Yuyang Wanyan, Shengze Shi, Yifei Song, Yue Duan, Shubo Lin, Jianshan Zhao, En Lin, Shiyin Lu, Qing-Guo Chen, Zhao Xu, Weihua Luo

llms computer-vision reinforcement-learning efficiency multimodal benchmarks

arXiv Computer Vision

Oct 27

KINESYS: Test-Time Adaptation for Reference-Based Effect Editing in Images

arXiv:2610.33292v1 Announce Type: new Abstract: Visual effects are often easier to demonstrate via reference than to describe in words. We study effect-reference editing, where a reference image show...

By Noah Frahm, Luchao Qi, Haoze Li, Dan B Goldman, Sambit Bhattacharya, Roni Sengupta

fine-tuning multimodal benchmarks

arXiv Computer Vision

Oct 27

Tight Learned Event-Inertial Odometry (TLEIO): a Transformer-Based Architecture for End-to-End Motion from Event Streams

arXiv:2610.33356v1 Announce Type: new Abstract: Event cameras provide microsecond temporal resolution, high dynamic range, and sparse asynchronous measurements, making them compelling for odometry un...

By Matteo Cascone, David Jensen, Alessandro Pirini, Angelo Stefanini, Pietro Ven\`e, Marta Tintor\'e Gazulla, Javier Hidalgo Carri\`o

llms benchmarks

arXiv Computer Vision

Oct 27

How Robust Are CNNs and Vision Transformers for Alzheimer's MRI Classification Across Datasets?

arXiv:2610.33416v1 Announce Type: new Abstract: Accurate assessment of Alzheimer's disease stages from MRI is important for understanding disease progression and supporting the development of compute...

By Hafija Aktar, Molly Croes, Frank S. Menniti, Abdeltawab Hendawi

llms benchmarks

arXiv Computer Vision

Oct 27

What Remains OOD? OODConceptBench for Open-World Concept Recognition in Vision-Language Models

arXiv:2610.33564v1 Announce Type: new Abstract: In the era of large-scale pretrained models, zero-shot generalization has increasingly blurred the boundary between known and unknown concepts. However...

By Zhiran Luo, Zhiyong Yang, Zitai Wang, Sicong Li, Qianqian Xu, Qingming Huang

llms multimodal benchmarks

arXiv Computer Vision

Oct 27

CADonatello: Towards Real-World Multi-Part 3D CAD Reconstruction

arXiv:2610.33575v1 Announce Type: new Abstract: Existing methods for CAD reconstruction from 3D scans achieve strong performance on standard benchmarks, yet their effectiveness often degrades in real...

By Ahmet Serdar Karadeniz, Dimitrios Mallis, Parsa Akbari Chianeh, Kseniya Cherenkova, Anis Kacem, Djamila Aouada

computer-vision fine-tuning benchmarks

arXiv Computer Vision

Oct 27

LENS: Likelihood Evaluation and Neighbor Smoothing for Attribute-Based Person Retrieval on Unseen Cameras

arXiv:2610.33641v1 Announce Type: new Abstract: Attribute-based person retrieval ranks surveillance images from a binary query of semantic attributes, but mean Average Attribute Degree of Match (mADM...

By Nguyen Dang Duy Tan, Vo Thanh Nguyen, Linh T. T. Tran, Huynh Nguyen Phat

fine-tuning benchmarks

Get the daily briefing