Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv AI

Sep 10

Qiushi Engine on AstaBench E2E-Bench-Hard

arXiv:2609.08196v1 Announce Type: new Abstract: This report analyzes Qiushi Engine v0.8 across all 40 test tasks in AstaBench E2E-Bench-Hard, a benchmark that requires autonomous agents to carry a re...

By Wenhao Li, Shuxing Yang, Fujia Chen, Jincheng Mi, Yuang Pan, Rui Zhao, Zichen Li, Junyao Wu, Shenzhan Hong, Yaqi Li, Yize Wang, Kaihao Zhu, Taowen Deng, Junjie Yang, Hongsheng Chen, Yihao Yang

agents benchmarks

arXiv Statistics ML

Sep 10

Variance-Reduced Fast Krasnoselkii-Mann Methods for Finite-Sum Root-Finding Problems

arXiv:2406.02413v4 Announce Type: replace-cross Abstract: We propose a new class of fast Krasnoselkii--Mann methods with variance reduction to solve a finite-sum co-coercive equation $Gx = 0$. Our al...

By Quoc Tran-Dinh

benchmarks

arXiv Machine Learning

Sep 10

Stochastically Perturbed Weights: Ensembles from Deterministic Machine-Learning Weather Models

arXiv:2609.08412v1 Announce Type: new Abstract: Machine-learning weather models (MLWMs) now match or outperform operational numerical weather prediction (NWP) at global medium-range forecasting, at f...

By Simon Adamov, Oliver Fuhrer, Reto Knutti, Sebastian Schemm

benchmarks

arXiv Machine Learning

Sep 10

SIM: Subspace Interaction-based Method for Token-Level Text Anomaly Detection

arXiv:2609.08200v1 Announce Type: new Abstract: Token-level text anomaly detection, as an emerging trend of text anomaly detection, moves beyond coarse-grained document-level detection by localizing...

By Kehan Yan, Yue Tan, Qingfeng Chen, Shiyuan Li, Yu Zheng, Yixin Liu

llms rag benchmarks safety

arXiv AI

Sep 10

One MLLM, One Call: Efficient Zero-Shot Vision-and-Language Navigation via Spatial-Aware Waypoints

arXiv:2609.06476v1 Announce Type: cross Abstract: Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires an embodied agent to navigate unseen environments by following natural la...

By Shiqi Pan, Qi Zheng, Hanqin Sun, Youjian Zhang, Daquan Feng, Xu Wang

agents robotics efficiency benchmarks

arXiv AI

Sep 10

SRD-GUARD: A Defense Framework of LLMs via Semantic Rewriting and Joint Multi-Model Scoring for Latent Intent Exposure

arXiv:2609.06540v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed in safety-critical applications, yet jailbreak attacks can conceal harmful intent through role...

By Qi Wang, Chengcheng Wan, Jiangtao Wang

llms benchmarks safety

arXiv Machine Learning

Sep 10

Target-Independent Micro-Interventions for Predicting Training Response Across Language-Model Families

arXiv:2609.08618v1 Announce Type: new Abstract: Benchmark scores describe what a checkpoint can do now, but they do not determine how it will respond to the next training episode. We measure this mis...

By Zhongxuan Liu, Sicheng Zhou, Hongzhi Wang

benchmarks

arXiv Machine Learning

Sep 10

HOPE: Heterophily-Aware Open-Set Node Classification with Pseudo-Extrapolation

arXiv:2609.08685v1 Announce Type: new Abstract: Standard open-set node classification methods rely on the homophily assumption, where connected nodes share labels. However, real-world graphs are ofte...

By Yumeng Dai, Yue Tan, Yixin Liu, Chenxu Wang, Pinghui Wang, Tao Qin

benchmarks

arXiv Computer Vision

Sep 10

Video-MOPD: Multi-Teacher On-Policy Distillation for Video Understanding

arXiv:2609.09300v1 Announce Type: new Abstract: Video understanding demands a convergence of complementary capabilities across perception, temporal understanding, and complex reasoning, which are dif...

By Zhenxin Qin, Peng Shi, Cong Han, Yinlong Qian, Zequn Jie, Lin Ma

reinforcement-learning efficiency benchmarks

arXiv Computer Vision

Sep 10

Pretraining and Distillation Matter More Than Architecture Family for Label-Free Single-Cell Classification

arXiv:2609.09863v1 Announce Type: new Abstract: Choosing a deep learning architecture for label-free single-cell classification remains an open question, with microscopy benchmarks reporting conflict...

By Philip Graemer, Giuseppe Di Caprio

llms computer-vision fine-tuning efficiency benchmarks

arXiv Computation and Language

Sep 10

CityPlanner: A Sandbox Agent for Executable Urban Planning

arXiv:2609.09578v1 Announce Type: cross Abstract: Urban planning is a real-world spatial optimization problem that requires selecting feasible actions from large candidate spaces under practical obje...

By Wentao Zhang, Jingyuan Wang, Zetong Zhou, Yifan Yang, Wenrui Wang

llms agents reinforcement-learning benchmarks

arXiv Computation and Language

Sep 10

OntologyAligner: Ontology-Aligned Retrieval and Hierarchy-Guided Large Language Model Reranking for Biomedical Ontology Normalization

arXiv:2609.10055v1 Announce Type: cross Abstract: Biomedical ontology normalization maps free-text expressions to standardized concepts, enabling consistent integration and analysis of biomedical dat...

By Jie Song, Zhichuan Xu, Ziyu Lu, Meng Xiao, Cheng Bi, Yuxin Zhang, Xin Zheng, Xiaoran Li, Qiongfang Cao, Hao Yang, Bairong Shen

llms rag benchmarks

arXiv Computation and Language

Sep 10

RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases

arXiv:2609.10092v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly act as research agents, yet their ability to track shifts in research attention is difficult to evaluate be...

By Yingqian Wu, Jingcong Liang, Siyuan Wang, Zhenfei Yin, Philip Torr, Junchi Yu, Zhongyu Wei

llms agents fine-tuning benchmarks

arXiv Computation and Language

Sep 10

Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts

arXiv:2609.09241v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large f...

By Dohyeon Kim, Bedionita Soro, Sung Ju Hwang

llms benchmarks safety

arXiv Computation and Language

Sep 10

From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Calls

arXiv:2609.09476v1 Announce Type: cross Abstract: In-vehicle assistants must translate natural-language requests into accurate vehicle function calls under strict memory and latency constraints, maki...

By Hamed Jafarzadeh Asl, Yuanhao Yu, Vahid Partovi Nia

llms agents fine-tuning benchmarks

arXiv Computer Vision

Sep 10

Myocardial Strain Drift Correction in Deep Learning Based Ultrasound Tracking

arXiv:2609.09577v1 Announce Type: cross Abstract: Myocardial strain from echocardiography is a key biomarker for cardiac function. Recent deep learning methods show strong performance for myocardial...

By Thierry Judge, Nicolas Duchateau, Andreas {\O}stvik, Havard Dalen, Bj{\o}rnar Grenne, Pierre-Yves Courand, Lasse Lovstakken, Pierre-Marc Jodoin, Olivier Bernard

fine-tuning benchmarks

arXiv Computer Vision

Sep 10

Hyperbolic Geometry for Open-World Object Detection in Remote Sensing Imagery

arXiv:2609.09626v1 Announce Type: new Abstract: Open-world object detection (OWOD) extends closed-set detection by requiring models to identify unknown objects and incrementally learn them once annot...

By Wuzhou Li, Jiawei Zhou, Shenghang Wang, Xiang Li

rag computer-vision benchmarks

arXiv Computation and Language

Sep 10

Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling

arXiv:2609.09691v1 Announce Type: new Abstract: When training data are limited, increasing parameter count is not the only way to improve language-model performance. A small parameter set, when repea...

By Tingshuo Fan, Hongtao Mu, Tianyu Zhou, Hansen Liu, Tao Ji

llms nlp benchmarks

arXiv Computer Vision

Sep 10

Distilling Image Prototypes for Guided Test-Time Adaptation

arXiv:2609.09737v1 Announce Type: new Abstract: Test-Time Adaptation (TTA) enhances the robustness of models against distribution shifts but faces two critical challenges: error accumulation from noi...

By Liwen Wang, Xingbo Dong, Iman Yi Liao, Deyin Liu, Massimo Tistarelli, Lin Yuanbo Wu, Zhe Jin

benchmarks

arXiv Computation and Language

Sep 10

Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?

The paper addresses the challenge of long input contexts in Retrieval-Augmented Generation (RAG) systems, where concatenating many retrieved chunks increases prefill workload and time to first token (TTFT). It proposes a dual strategy: fine‑tuning the model to be aware of KV cache concatenation and selectively recomputing only part of the KV caches. Experiments on the RULER benchmark show that for a 124k‑token input, this combined method boosts the RULER score by 9.7 points over a baseline that recomputes caches only, while cutting TTFT by 80% compared with full attention.

By Fumihiko Tachibana, Daisuke Miyashita, Jun Deguchi

rag fine-tuning efficiency benchmarks

Get the daily briefing