Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

12,573 stories · RSS feed

arXiv Machine Learning
Aug 13

NAE: Normalizing AutoEncoder

arXiv:2608. 12084v1 Announce Type: new Abstract: We consider the setting of Normalizing flows with approximate inverses, an established paradigm spanning both full-dimensional ($d=D$) and bottleneck ($d<D$) settings, and group these models under the term flow autoencoders.

By Muhammad Abdur Rafae, Niels Landwehr
arXiv Machine Learning
Aug 13

Automated binary classification of hazelnut X-ray images: A deep-learning benchmark for quality assessment

arXiv:2608. 11759v1 Announce Type: cross Abstract: Non-destructive X-ray imaging can reveal internal hazelnut defects that are difficult to detect by external inspection alone; however, automated interpretation remains challenging because of subtle radiographic differences among classes, marked class imbalance, and limited annotated data.

By Giancarlo Sportelli, Nicola Belcari, Roberta Pace, Umberto Bernardo, Sharmin Sultana, Alessandra Toncelli, Matteo Giaccone
arXiv AI
Aug 13

CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

arXiv:2608. 12002v1 Announce Type: new Abstract: Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints.

By Xingyu Yan, Tingting Dai, Antonio De Domenico, Mohamed Sana, Nicola Piovesan, Changchang Li, Bowen Liu, Kun Jiang, Mengjie Zhang, Dingcheng Shan, Jing-Cheng Pang, Chenwei Wu, Sijie Wu, Lianying Chao, Haoran Cai, Jiantao Ye, Xubin Li, Simon Mark Lucas, Xin Chen
arXiv AI
Aug 13

Reinforcement Learning based DBMS Buffer Pool Auto-Tuning for Optimal Memory Utilization

arXiv:2608. 11239v1 Announce Type: cross Abstract: Administering Database Management Systems (DBMS) instances requires Database Administrators (DBA) to balance performance in terms of Service Level Agreement (SLA) against resource usage, often prompting RAM over-allocation that wastes memory.

By Yifan Wang, Patrick Royer, Rapha\"el F\'eraud, David Delande
arXiv AI
Aug 13

Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability

arXiv:2608. 11238v1 Announce Type: new Abstract: Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across the diverse spectrum of user queries, ranging from close-ended fact-seeking to open-ended explanatory requests.

By Jeonghwan Choi, Taewon Yun, Minjeong Ban, Gyeonghun Sun, Jae-Gil Lee, Hwanjun Song