Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI

Sep 23

Compared to What? A Human-Anchored Security Benchmark for LLM-Generated Infrastructure-as-Code

arXiv:2608.28021v2 Announce Type: replace-cross Abstract: Large language models increasingly author Infrastructure-as-Code (IaC), where one insecure default is provisioned straight into production. P...

By Animesh Shaw

llms benchmarks

arXiv Machine Learning

Sep 23

Entropy Can Flow, or It Can Guide. Be Entropy. LEDFlow: Introducing Entropy-guided Generation Order into Uniform Discrete Flow

arXiv:2609.25131v1 Announce Type: new Abstract: Uniform discrete flow permits repeated updates at every generation position. While continued revision supports correction of wrong tokens, it also expo...

By Tung Sum Thomas Kwok, Yidong Ouyang, Yingjia Wan, Ying Nian Wu, Zhijiang Guo, Oscar Leong

diffusion multimodal benchmarks

arXiv Machine Learning

Sep 23

Exposing Blind Spots in Deep Imbalanced Regression Evaluation

arXiv:2609.25152v1 Announce Type: new Abstract: Deep Imbalanced Regression (DIR) addresses a common failure mode of regression models: target distributions are highly non-uniform, causing models to p...

By Noah C. Puetz, Jens U. Brandt, Marc Hilbert, Elena Raponi, Thomas B\"ack, Thomas Bartz-Beielstein

multimodal benchmarks

arXiv Machine Learning

Sep 23

Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers

arXiv:2609.25237v1 Announce Type: new Abstract: Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned,...

By Weihang Ding, Junfei Zhan

llms agents fine-tuning benchmarks

arXiv Machine Learning

Sep 23

A JEPA Recipe for Tabular Foundation Models

arXiv:2609.25541v1 Announce Type: new Abstract: Tabular foundation models learn to predict cell values in context, whereas world-model self-supervision asks for prediction in representation space (Le...

By Mingyu Jeon, Suwan Cho, Jae Young Suh

rag benchmarks

arXiv Machine Learning

Sep 23

Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indices

arXiv:2609.25645v1 Announce Type: new Abstract: Exhaustively evaluating every candidate LLM configuration on every benchmark item to identify a high-performing one is costly. We formulate configurati...

By Qian Xie, Yueli He, Nairen Cao

llms reinforcement-learning benchmarks

arXiv Machine Learning

Sep 23

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

arXiv:2609.25655v1 Announce Type: new Abstract: As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectu...

By Zhentao Tan, Chang Liu, Yao Liu, Yue Wu, Jieping Ye

llms fine-tuning benchmarks

arXiv Machine Learning

Sep 23

Graph Domain Adaptation Does Not End with Representation Learning

arXiv:2609.25692v1 Announce Type: new Abstract: Graph domain adaptation (GDA) transfers knowledge from a labeled source graph to an unlabeled target graph under shifts in both node attributes and gra...

By Ziqian Liu, Yongxue Xu, Enze Zhang, Jiaqi Zhang, Hao Wang, Maolin Wang

fine-tuning benchmarks safety

arXiv Machine Learning

Sep 23

Disentangling Heterogeneous Traffic Dynamics for Multi-Step Traffic Forecasting via Adaptive Spectral Decomposition

arXiv:2609.25777v1 Announce Type: new Abstract: Accurate multi-step traffic forecasting remains challenging because observed traffic signals contain heterogeneous temporal dynamics with different cha...

By Zijun Huang, Chenrui Fu, Wenhao Wang, Xiaochuan Gou, Chih-Chieh Hung, Guanyao Li

benchmarks

arXiv Machine Learning

Sep 23

Evaluating Accuracy and Probabilistic Reliability of Zero-Shot Time Series Foundation Models

arXiv:2609.25788v1 Announce Type: new Abstract: Time Series Foundation Models (TSFMs) promise a paradigm shift toward zero-shot forecasting by eliminating task-specific training. However, existing wo...

By Panagiotis Michael, Moysis Symeonides, Demetris Trihinas

llms benchmarks

arXiv Machine Learning

Sep 23

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

arXiv:2609.25809v1 Announce Type: new Abstract: Fine-grained mixture-of-experts (MoE) architectures have become a mainstream design for open-weight LLMs, with hundreds of experts and increasingly man...

By Yuanteng Chen, Qiwei Lai, Chen Tianqi, Peisong Wang, Yuantian Shao, Nanxin Zeng, Zhilei Liu, Chuangyi Li, Jing Liu, Jian Cheng

llms efficiency multimodal benchmarks

arXiv Machine Learning

Sep 23

In-Context Guidance: Learning Inter-Task Synergies via Numerical Foundational Models for Few-Shot Multitask Optimization

arXiv:2609.25836v1 Announce Type: new Abstract: Multi-task optimization (MTO) addresses a set of optimization tasks simultaneously, often suffering from inaccurate inter-task relationship estimation...

By Tingyang Wei, Haofeng Wu, Jiao Liu, Zhao Wei, Puay Siew Tan, Yew-Soon Ong

llms robotics benchmarks

arXiv Machine Learning

Sep 23

Evaluating the Effectiveness of SechKAN on 1D Data

arXiv:2609.25876v1 Announce Type: new Abstract: The connection between the Kolmogorov-Arnold representation theorem (KART) and neural network design has led to the development of Kolmogorov-Arnold Ne...

By Hoang-Thang Ta

benchmarks

arXiv Machine Learning

Sep 23

Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL

arXiv:2609.25938v1 Announce Type: new Abstract: A conformal abstention certificate for text-to-SQL is only as truthful as the correctness labels it is calibrated on. The uncertainty pipelines that re...

By Jiamiao Liu, Dewen Qiao, Yu Zhang, Xuetao Chen

benchmarks data-engineering

arXiv Machine Learning

Sep 23

GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

arXiv:2609.25963v1 Announce Type: new Abstract: Transformer architectures exhibit cross-layer redundancies, yet post-training compression pipelines typically optimize layers in isolation or rely on h...

By Baher Mohammad, Ammar Ali, Stamatios Lefkimmiatis

llms efficiency benchmarks

arXiv Machine Learning

Sep 23

FuncCode: Compressing Kolmogorov--Arnold Networks in Function Space with Hardware-Aware Quantization

arXiv:2609.26067v1 Announce Type: new Abstract: Kolmogorov--Arnold Networks (KANs) replace scalar edge weights with learnable univariate functions, increasing flexibility but also parameter memory be...

By Kazi Ahmed Asif Fuad, Lizhong Chen

efficiency benchmarks

arXiv Machine Learning

Sep 23

Block-Level Weight-Space Structure Persists Under Post-Training: An Empirical Study Across LLM Families

arXiv:2609.26147v1 Announce Type: new Abstract: Modern LLMs are deployed as families of post-trained variants (base, instruct, chat, code) derived from a shared set of pre-trained weights. We present...

By Zhaohui Wang

llms benchmarks

arXiv Machine Learning

Sep 23

Information-Theoretic Decoupled Prompt Tuning for Continual Learning

arXiv:2609.26257v1 Announce Type: new Abstract: Continual learning (CL) aims to incrementally acquire knowledge from sequential data while avoiding catastrophic forgetting. Recently, prompt tuning ha...

By Yunfei Zhang, Wen Wen, Tieliang Gong, Weizhan Zhang

benchmarks safety

arXiv Machine Learning

Sep 23

JAMPR+/L2D: scalable neural heuristic for constrained vehicle routing problems in dynamic environment

arXiv:2609.26275v1 Announce Type: new Abstract: The vehicle routing problems with real-world constraints (we consider vehicles capacity limits, time windows constrains, pickup-and-delivery multi-depo...

By Andrew Soroka, Alex Meshcheryakov

reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Sep 23

Disaggregated Quantization: Specializing LLM Prefill and Decode

arXiv:2609.26333v1 Announce Type: new Abstract: Prefill and decode reward different approaches to quantization: low-precision arithmetic accelerates prompt processing, while compact weights reduce me...

By Andrei Panferov, Maximilian Kleinegger, Sweta Priyadarshi, Tijmen Blankevoort, Dan Alistarh

llms efficiency benchmarks

Get the daily briefing