Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computer Vision

Oct 27

A Sentinel-2 benchmark dataset for deep-learning active-fire segmentation across 25 California wildfires, July 2020-August 2026

arXiv:2609.16199v2 Announce Type: replace-cross Abstract: This article describes an open image dataset for developing and evaluating active-fire segmentation methods in satellite imagery. The dataset...

By Shreyan Mitra, Mohammadreza Narimani, Parastoo Farajpoor

computer-vision benchmarks

arXiv Statistics ML

Oct 27

Identities, Not Pairs: Distribution-Free False-Accept Control for Embedding-Based Verification

arXiv:2610.35277v1 Announce Type: new Abstract: Embedding-based verification often calibrates a false-accept threshold from all impostor pairs among $m$ identities, although these pairs share identit...

By Salim Khazem

rag benchmarks

arXiv Statistics ML

Oct 27

Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

arXiv:2610.35514v1 Announce Type: new Abstract: Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the expone...

By Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae

benchmarks

arXiv Statistics ML

Oct 27

Graph Neural Networks for Supply Chain Management: A Framework and Real World Benchmark

arXiv:2411.08550v3 Announce Type: replace-cross Abstract: Graph Neural Networks (GNNs) have recently gained traction in transportation, bioinformatics, language and image processing, but research on...

By Azmine Toushik Wasi, MD Shafikul Islam

benchmarks

arXiv Statistics ML

Oct 27

Lumbermark: Resistant Clustering by Chopping Up Mutual Reachability Minimum Spanning Trees

arXiv:2604.07143v2 Announce Type: replace-cross Abstract: We introduce Lumbermark, a robust divisive clustering algorithm capable of detecting clusters of varying sizes, densities, and shapes. Lumber...

By Marek Gagolewski

benchmarks

arXiv AI

Oct 27

Scaling Qualitative Analysis with Large Language Models: Development and Validation of the Smart Coder Framework for Human-LLM Collaborative Coding

arXiv:2610.33205v1 Announce Type: new Abstract: Current uses of large language models (LLMs) for qualitative coding often rely on ad hoc prompting with limited guidance for reliability and human over...

By Cheng Tang, Kun Wang, Yaxuan Yang, Shiyu Wang, Allan Cohen, Rachael Eriksen Brown, Chandra Hawley Orrill

llms benchmarks

arXiv AI

Oct 27

NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting

arXiv:2610.33240v1 Announce Type: new Abstract: As frontier AI systems increasingly serve as sources of political and current-events information, evaluating the epistemic quality of their responses i...

By Matt Wilde, Jillian Fisher, Andrew B. Hall, Kathryn Salam, Robbie Goldfarb

benchmarks safety

arXiv AI

Oct 27

Conformance as a Distribution: Delegation Contracts, Error Budgets, and the Allocation of Evaluation Authority in Use-Based AI Regulation

arXiv:2610.33112v1 Announce Type: new Abstract: Use-based AI regulation assigns obligations by deployment risk but lacks an evidentiary object to carry them. A benchmark score is a property of a mode...

By John deVadoss

llms benchmarks

arXiv AI

Oct 27

Auditing evaluation artifacts and data leakage in large language model systems for rare disease diagnosis

arXiv:2610.33592v1 Announce Type: new Abstract: Augmented large language model (LLM) systems have reported high benchmark accuracies in rare disease diagnosis, yet their real-world clinical utility r...

By Jian Yang, Peilin Liu, Xiaohan Li, Songda Li, Yu Cao, Yuqing Feng, Haomin Li

llms agents benchmarks safety

arXiv AI

Oct 27

Beyond Static Structures: Linking Protein Dynamics to Function via DynaReason-Pro

arXiv:2610.33804v1 Announce Type: new Abstract: Intrinsically disordered proteins (IDPs) carry out their functions through heterogeneous conformational ensembles that reflect their dynamics, yet most...

By Zhaokang Liang, Mingze Yuan, Quanzheng Li, Wengong Jin

benchmarks

arXiv AI

Oct 27

From Document Layout to Causal Topology: An End-to-End Architecture for Temporal Causal Reasoning

arXiv:2610.34094v1 Announce Type: new Abstract: Large language models (LLMs) generally lack deep causal reasoning capabilities in temporal reasoning. Inspired by document layout reordering, this pape...

By Zhifeng Yang, Wei Luo, Yongjian Chen, Shiqi Ou

llms agents benchmarks

arXiv AI

Oct 27

Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents

arXiv:2610.34167v1 Announce Type: new Abstract: Tool-using language model agents are usually evaluated by whether they finish the user's task. That metric misses a common failure: an agent can comple...

By Jiaju Huang, Shaobin Chen, Xinglong Liang, Xinyu Ma, Hao Yang, Wei Ke, Tao Tan, Yue Sun

llms agents benchmarks

arXiv AI

Oct 27

Agentic AI Orchestration Framework for Spatial Intelligence: Supply Chain Deployment

arXiv:2610.34569v1 Announce Type: new Abstract: Industrial spatial decisions are assembled across systems that share neither a reference frame nor an owner, and analysts spend substantial effort reco...

By Constantinos Papantoniou, Brian Hilton

agents benchmarks data-engineering

arXiv AI

Oct 27

HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget

arXiv:2610.34582v1 Announce Type: new Abstract: We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rent...

By Harish Santhanalakshmi Ganesan, Hariharan M. S., Prabhurajhan R

llms agents diffusion benchmarks

arXiv AI

Oct 27

Movable-Element Multi-Functional RIS-Assisted Hybrid RSMA-NOMA: A Constrained Hybrid Reinforcement Learning Approach

arXiv:2610.34701v1 Announce Type: new Abstract: In this work, we propose a movable-element (ME) multi-functional reconfigurable intelligent surface (MF-RIS) architecture, termed MM-RIS, which integra...

By Li-Hsiang Shen, Chi-Te Kuo

llms agents reinforcement-learning benchmarks

arXiv AI

Oct 27

The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory

arXiv:2610.34767v1 Announce Type: new Abstract: Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific...

By Yuexi Pan, Yongchao Chen

agents benchmarks

arXiv AI

Oct 27

Collective Emergent Alignment: Measuring and Amplifying Interaction-Induced Reasoning Alignment in Multi-Agent LLMs

arXiv:2610.34836v1 Announce Type: new Abstract: Multi-agent large language model (LLM) systems can exhibit emergent behaviors through interaction, yet whether a similar phenomenon exists for reasonin...

By Boxuan Wang, Zhuoyun Li, Sarvapali D. Ramchurn, Xiaowei Huang, Yi Dong

llms agents benchmarks safety

arXiv AI

Oct 27

UniGFM-RAG: Unify Graph Foundation Models and Large Language Models for End-to-End Retrieval and Reasoning

arXiv:2610.34866v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown remarkable capabilities across tasks, but they still struggle with complex reasoning that requires integrating...

By Zicheng Zhao, Linhao Luo, Zongji Yu, Junnan Dong, Yanchao Tan, Shirui Pan, Chen Gong

llms rag efficiency benchmarks

arXiv AI

Oct 27

C-SafeGen: Certified Safe LLM Generation with Claim-Based Streaming Guardrails

arXiv:2610.34961v1 Announce Type: new Abstract: Despite the remarkable capabilities of large language models (LLMs) across diverse applications, they remain vulnerable to generating content that viol...

By Mintong Kang, Zhaorun Chen, Bo Li

llms benchmarks safety

arXiv AI

Oct 27

Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs

arXiv:2610.35069v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly evaluated for fairness, yet most existing benchmarks audit one sensitive attribute at a time. This single...

By Zheng Zhang, Yifan Wu, Yuxiang Zhou, Qi Liu, Qingyang Mao, Ning Li, Weibo Gao, Qiyuan Shi, Zhenya Huang, Baosheng Yu

llms benchmarks safety

Get the daily briefing