Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI

Sep 30

MCP Error Messages Written for Developers Hurt the Most Capable Agents Most

arXiv:2609.35381v2 Announce Type: replace-cross Abstract: Many Model Context Protocol (MCP) servers wrap web APIs built for human developers, and their error messages tell the reader to run a command...

By Xiaonan Xu, Wenjing Wu

llms agents benchmarks

arXiv Machine Learning

Sep 30

Serverless gossip training of LSTM failure detectors: A matched-protocol comparison with federated, local and centralized learning on NASA C-MAPSS

arXiv:2609.35792v1 Announce Type: new Abstract: Industrial predictive maintenance increasingly depends on learning from equipment spread across sites whose sensor data cannot easily be pooled. Federa...

By Yusuf \"Ozt\"urk, Enes G\"oktekin, Bengisu Atl{\i}, Ak{\i}n \"Ozt\"urk, Zhixiang Wang, Ulas Bagci

benchmarks

arXiv Machine Learning

Sep 30

CADENCE: A Confidence-Adaptive Dual-Expert Network for Fast and Accurate Time Series Classification

arXiv:2609.35929v1 Announce Type: new Abstract: Time series classification (TSC) exhibits a sharp trade-off between accuracy and computational scalability. Meta-ensembles like HIVE-COTE 2.0 reach sta...

By Onisa Mpaunda

benchmarks

arXiv Machine Learning

Sep 30

Preferent Compression Bounds Are Tight

arXiv:2609.36030v1 Announce Type: new Abstract: The lack of rigorous safety and performance certificates remains a key bottleneck to the deployment of modern learning-based methods. Sample compressio...

By Dario Paccagnan, Marius Tirlea

benchmarks safety

arXiv Machine Learning

Sep 30

Koa-action: Fast and Consistent Structured Decision Making with Generative LLMs

arXiv:2609.36115v1 Announce Type: new Abstract: Industry applications often demand low-latency classification, yet current large language model (LLM) approaches remain poorly suited for latency-criti...

By Shenghong Dai, Shiva Kumar Pentyala, Yingchi Liu, Shubham Mehrotra, Suman Banerjee, James Zhu, Bin Bi, Sitaram Asur, Phil Mui

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Sep 30

Fractional State Space Transition for Long Sequence Modeling

arXiv:2609.36314v1 Announce Type: new Abstract: State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for...

By Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi, Lifeng Shang, Yufei Cui

benchmarks

arXiv Machine Learning

Sep 30

AdaptArena: Evaluating Test-Time Personalization of Web Agents

arXiv:2609.36488v1 Announce Type: new Abstract: Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings...

By Dongchan Shin, Xing Han L\`u, Jiaqi Deng, Jay Gala, Tom\'as Vergara Browne, Jaewon Moon, Fengyuan Liu, Alexandre Drouin, Siva Reddy, Alexandre Lacoste

llms agents nlp benchmarks

arXiv Machine Learning

Sep 30

PDE-OBS: Controlled Evaluation Across Observation Patterns

arXiv:2609.36521v1 Announce Type: new Abstract: Physical-field reconstruction and forecasting depend on both measurement density and spatial layout, yet evaluation under a single observation pattern...

By Ruichen Xu, Siyao Wang, Fang Wan, Jiacheng Qiu, Wenhan Gao, Jiaxing Zhang, Linsey Pang, Ravid Shwartz-Ziv, Yann LeCun, Yuefan Deng

benchmarks

arXiv Machine Learning

Sep 30

Human-inspired, Task-Dimension-Guided Exploration for Efficient Learning in High Dimensions

arXiv:2609.36672v1 Announce Type: new Abstract: Efficient exploration in high-dimensional decision spaces remains a central challenge for decision-making systems. Humans, in contrast, can navigate la...

By Fanyu Zhu, Jiahui An, Ni Ji

benchmarks

arXiv Machine Learning

Sep 30

Where Root Cause Analysis Fails: A Retrieval-Reranking Decomposition

arXiv:2609.36686v1 Announce Type: new Abstract: Identifying the root cause of an anomaly among hundreds of sensors is critical for preventing safety incidents and costly downtime in complex monitored...

By Hada Melino Muhammad, Luan Pham, Laure Barri\`ere, Sachin Shetty, Leonardo Pulga, Flora D. Salim

llms rag benchmarks safety

arXiv Machine Learning

Sep 30

cktFormer: Transformer-Based Approach for Automated Analog Circuit Design

arXiv:2609.36752v1 Announce Type: new Abstract: Circuit design is a complex and iterative process that requires expertise in electronic engineering. It involves selecting components while meeting per...

By Pasindu Dodampegama, Praveen Wijesinghe, Naveen Basnayake, Keshawa Jayasundara, Tharindu Bandaragoda

llms benchmarks

arXiv Machine Learning

Sep 30

Graph-Spectral Flow Matching for Multivariate Time Series Anomaly Detection

arXiv:2609.36765v1 Announce Type: new Abstract: Multivariate time series anomaly detection typically relies on evaluating discrepancies between observations and outputs produced by models trained on...

By Zepeng Zhang, Jhony H. Giraldo, Wenbin Wang, Olga Fink

diffusion benchmarks

arXiv Machine Learning

Sep 30

Scalable Diffusion SBI for Compositional Inference under Simulator Misspecification

arXiv:2609.36950v1 Announce Type: new Abstract: Simulation-based inference is challenging when many heterogeneous observations must be composed, hierarchical latent structure must be preserved, and t...

By Vincent D. Zaballa, Elliot E. Hui

diffusion fine-tuning benchmarks

arXiv Machine Learning

Sep 30

Equally Good, Yet Different: Benchmarking Rashomon sets in AutoML packages

arXiv:2609.36970v1 Announce Type: new Abstract: The Rashomon effect describes the existence of multiple near-optimal models that achieve comparable performance while offering fundamentally different...

By Katarzyna Wo\'znica, Katarzyna Rogalska, Zuzanna Sie\'nko, Mustafa Cavus

benchmarks

arXiv Machine Learning

Sep 30

Rethinking Soft Tokens for Parallel Decoding in Diffusion Language Models

arXiv:2609.37391v1 Announce Type: new Abstract: Diffusion language models (DLMs) enable parallel generation by predicting and committing multiple tokens at each denoising step, yet they can generate...

By Kodai Kawamura, Kenji Kawaguchi, Anji Liu

llms rag diffusion benchmarks

arXiv Machine Learning

Sep 30

REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse

arXiv:2609.37500v1 Announce Type: new Abstract: On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its relianc...

By Yuxiao Yang, Shangzhe Li, Tianrun Yu, Kaixiang Zhao, Taylor W. Killian, Weitong Zhang

llms efficiency benchmarks

arXiv Machine Learning

Sep 30

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

arXiv:2609.37616v1 Announce Type: new Abstract: Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on th...

By Abhinav Rajeev Kumar (Lossfunk), Paras Chopra (Lossfunk)

llms benchmarks safety

arXiv Machine Learning

Sep 30

TabFM: A Zero-Shot Foundation Model for Tabular Data

arXiv:2609.37959v1 Announce Type: new Abstract: Tabular machine learning typically relies on per-dataset workflows, fitting tree ensembles or running AutoML searches from scratch for every task. We p...

By Weihao Kong, Erez Louidor Ilan, Shuxin Nie, Taman Narayan, Rajat Sen, Yichen Zhou, Deqing Fu, Samet Oymak, Abhimanyu Das

llms benchmarks

arXiv Machine Learning

Sep 30

Prompts Live on an Arc: Gaussian Curricula in Fisher--Rao Coordinates for Rollout-Efficient GRPO

arXiv:2609.38018v1 Announce Type: new Abstract: Group relative policy optimization (GRPO) learns only from prompts whose sampled responses disagree: a group that is entirely correct or entirely incor...

By Mei Okonkwo, Pixel Nomand, Julian Berg, Elena Voss, Lena Park, Marcus Hale, Adrian Cho, Sofia Reyes

benchmarks

arXiv Machine Learning

Sep 30

Multi-Class, Multi-Tier Network Intrusion Detection: A Comprehensive and Reproducible Benchmark

arXiv:2609.36039v1 Announce Type: cross Abstract: Machine learning (ML) and deep learning (DL) have dominated Intrusion Detection System (IDS) research in recent years. Unfortunately, many existing s...

By Yufeng Xin, Bryant Goseland, Mohamed Rahouti

benchmarks

Get the daily briefing