Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

13,478 stories · RSS feed

arXiv Machine Learning
Aug 13

Weightless Fine-Tuning: Personalizing LLMs via Logit-Space Transport

arXiv:2608. 11342v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) is a standard approach for adapting LLMs to a target distribution, but in settings such as personalization, where each author requires separate weight access, optimization, storage, and retraining, its costs become prohibitive.

By Bohan Zhang, Anqi Ni, Yixin Wang, Paramveer S. Dhillon
arXiv Machine Learning
Aug 13

NAE: Normalizing AutoEncoder

arXiv:2608. 12084v1 Announce Type: new Abstract: We consider the setting of Normalizing flows with approximate inverses, an established paradigm spanning both full-dimensional ($d=D$) and bottleneck ($d<D$) settings, and group these models under the term flow autoencoders.

By Muhammad Abdur Rafae, Niels Landwehr
arXiv Machine Learning
Aug 13

Automated binary classification of hazelnut X-ray images: A deep-learning benchmark for quality assessment

arXiv:2608. 11759v1 Announce Type: cross Abstract: Non-destructive X-ray imaging can reveal internal hazelnut defects that are difficult to detect by external inspection alone; however, automated interpretation remains challenging because of subtle radiographic differences among classes, marked class imbalance, and limited annotated data.

By Giancarlo Sportelli, Nicola Belcari, Roberta Pace, Umberto Bernardo, Sharmin Sultana, Alessandra Toncelli, Matteo Giaccone
arXiv AI
Aug 13

CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

arXiv:2608. 12002v1 Announce Type: new Abstract: Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints.

By Xingyu Yan, Tingting Dai, Antonio De Domenico, Mohamed Sana, Nicola Piovesan, Changchang Li, Bowen Liu, Kun Jiang, Mengjie Zhang, Dingcheng Shan, Jing-Cheng Pang, Chenwei Wu, Sijie Wu, Lianying Chao, Haoran Cai, Jiantao Ye, Xubin Li, Simon Mark Lucas, Xin Chen
arXiv AI
Aug 13

Reinforcement Learning based DBMS Buffer Pool Auto-Tuning for Optimal Memory Utilization

arXiv:2608. 11239v1 Announce Type: cross Abstract: Administering Database Management Systems (DBMS) instances requires Database Administrators (DBA) to balance performance in terms of Service Level Agreement (SLA) against resource usage, often prompting RAM over-allocation that wastes memory.

By Yifan Wang, Patrick Royer, Rapha\"el F\'eraud, David Delande