Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning

Sep 22

Leveraging Inference-Time Compute for Diffusion Models via Global Scheduling of Denoising Trajectories

The paper studies how to allocate a fixed computational budget across the denoising steps of diffusion models to improve sample quality at deployment. It shows that the expected benefit of evaluating multiple candidates at a step can be decomposed into a step‑specific sensitivity and a universal sample‑size factor, and that the optimal allocation follows a water‑filling structure. Experiments demonstrate that this allocation achieves the same quality as a uniform strategy while reducing function evaluations by 20–50%.

By Yuan Cao, Yifu Tang, Hangqi Li, Zeyu Zheng

diffusion benchmarks

arXiv Machine Learning

Sep 22

CSC: Calibrated Simplicity for Conflict-Aware Social Bot Detection in the LLM Era

The paper introduces CSC, a calibrated-simplicity framework for detecting social bots in the era of large language models. CSC combines a simplified prototype-guided graph expert, calibrated simplex-constrained fusion, and a lightweight inconsistency expert to address modality conflict between semantic and structural signals. Experiments on TwiBot-22, TwiBot-20, and MGStBot-large demonstrate that CSC improves calibrated decision quality while maintaining competitive performance across benchmarks.

By Yipeng Qian, Pengjie Zhao, Chaoxi Niu

llms robotics multimodal benchmarks

arXiv Machine Learning

Sep 22

ITSY: Causal Discovery From Irregular Time-Series Data

arXiv:2609.23516v1 Announce Type: new Abstract: Structural causal models for time series recover contemporaneous and lagged effects, but most methods require complete observation windows and become m...

By Wenbo Xu, Yue He, Yunhai Wang, Yueguo Chen, Kun Kuang

benchmarks

arXiv Machine Learning

Sep 22

ETH-TraceBench: A Large-Scale Event-Stream Benchmark for Ethereum DeFi under Temporal, Protocol, and Contract Shift

arXiv:2609.23659v1 Announce Type: new Abstract: Ethereum decentralized finance (DeFi) provides a public, time-stamped record of transaction-level event streams, but the same public symbols can create...

By Kemal Kirtac, Carsten Maple

llms benchmarks

arXiv Machine Learning

Sep 22

Falling Trees: A Model Class for Interpretable Risk Prioritization

arXiv:2609.23780v1 Announce Type: new Abstract: Many real-world decisions require prioritizing high-risk cases, such as clinicians prioritizing high-risk patients before lower-risk ones. Falling rule...

By Varun Babbar, Zachery Boner, Margo Seltzer, Cynthia Rudin

benchmarks safety

arXiv Machine Learning

Sep 22

Adaptive Determinantal Client Scheduling in Federated Learning

arXiv:2609.23843v1 Announce Type: new Abstract: Scheduling clients for model training is critical in federated learning due to both data and system heterogeneity. Most previous works focus on the qua...

By Wen Xu, Ben Liang, Gary Boudreau, Hamza Sokun

benchmarks

arXiv Machine Learning

Sep 22

Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering

arXiv:2609.24202v1 Announce Type: new Abstract: Sparse attention reduces the quadratic cost of global self-attention while retaining strong empirical performance, but how its restricted interactions...

By Jingkun Liu, Yue Song

llms benchmarks safety

arXiv Machine Learning

Sep 22

TTSE: A Two-Track Online Self-Evolution Framework

arXiv:2609.24289v1 Announce Type: new Abstract: As Large Language Model (LLM) agents are applied in continuously interactive environments, driving the evolution of their own capabilities becomes a co...

By Ruimin Pei, Yongkang Wu, Shangyi Zheng, Yaqing Zhang, Deyang Li, Jianjun Tao, Xinyu Zhang, Xiang Zhang

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Sep 22

Artificial Structure Function Search: Preserving Artificial Functional Connectivity for Structured Pruning

arXiv:2609.24401v1 Announce Type: new Abstract: Structured pruning is a model compression technique that is used to reduce the computational cost of deploying deep neural networks on resource-constra...

By Mindula Illeperuma, Rafael Pina, Charuka Herath, Sharmarke A. Gabayre, Varuna De Silva

efficiency benchmarks

arXiv Machine Learning

Sep 22

WPBench: A Comprehensive Benchmark for Wind Power Forecasting

arXiv:2609.24444v1 Announce Type: new Abstract: Accurate, reliable, and deployable wind power forecasting is critical for power system dispatch, renewable energy integration, and electricity market o...

By Yuhan Zhu, Jilin Hu, Xinying Cai, Yingshan Li, Li Ma, Xiangfei Qiu Linsen Li, Kai Zhang, Yao Fu, Weihao Jiang, Bin Yang

benchmarks

arXiv Machine Learning

Sep 22

RAILS: Retrieval-Augmented Incremental LLM Clustering at Scale

arXiv:2609.24464v1 Announce Type: new Abstract: Using a Large Language Model (LLM) as the clusterer at production scale is hard: prompts cannot hold the entire label space, and per-document serial pr...

By Armin Oliya, Aleksandra Sawczuk, Rados{\l}aw Bia{\l}obrzeski

llms rag efficiency benchmarks

arXiv Machine Learning

Sep 22

Overlay\_dx - Automating forecasting evaluation

arXiv:2609.24586v1 Announce Type: new Abstract: Traditional evaluation metrics provides numerical values but often lack comprehensibility, hindering effective differentiation of model performances. O...

By Long Ngo, Mohammed Amine Chamli, Jonathan Rivalan, Thomas Jaillon

benchmarks safety

arXiv Machine Learning

Sep 22

Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use

arXiv:2609.24985v1 Announce Type: new Abstract: Multi-turn tool-use failures can hinge on a single model call, yet reward variation alone does not reveal which call would benefit from training. When...

By Zixiang Chen, Wenting Zhao, Zhepeng Cen, Akshara Prabhakar, Jielin Qiu, Jianguo Zhang, Zhiwei Liu, Tulika Manoj Awalgaonkar, Liangwei Yang, Shelby Heinecke, Silvio Savarese, Huan Wang

agents reinforcement-learning benchmarks

arXiv Machine Learning

Sep 22

Leakage-Safe Empirical Benchmarking of EEG-Based Machine Learning Pipelines for Dementia Classification

arXiv:2609.22092v1 Announce Type: cross Abstract: Electroencephalography (EEG) is a low-cost and non-invasive signal source for dementia screening, yet existing EEG-based studies remain difficult to...

By Haitian Wang, Chamara Madarasingha, Redowan Mahmud, Aneesh Krishna, Ryu Takechi

computer-vision multimodal benchmarks

arXiv Machine Learning

Sep 22

Does the Truthfulness Signal Survive Code-Mixing? Probing Hidden States for Hallucination Detection in Hinglish

arXiv:2609.22138v1 Announce Type: cross Abstract: Hidden-state hallucination probing - training a linear classifier on an LLM's internal activations to detect whether a generated answer is faithful t...

By Tanveer Singh (Plaksha University)

llms benchmarks safety

arXiv Machine Learning

Sep 22

SCoR: A Hierarchical Framework for Forecasting Relations Between Scientific Concepts

arXiv:2609.22174v1 Announce Type: cross Abstract: Anticipating emerging research directions is a critical goal of AI-assisted science. Existing methods mainly predict which concepts will co-occur in...

By Jingze Wang, Fred Sun, Shangqi Guo

benchmarks

arXiv Machine Learning

Sep 22

EAVer: Long-Form Factuality Verification as an End-to-End Agentic Policy

arXiv:2609.22223v1 Announce Type: cross Abstract: Long-form factuality verification is commonly implemented as a static decompose-search-verify pipeline, with separately prompted modules processing c...

By Kening Zheng, Aoying Zheng, Zhigang Chang, Yazhi Guo, Miaotian Guo, Qingwei Zong, Xianhai Xie, Weiqiang Jin, Chengze Li, Hanrong Zhang, Jie Yang, Wei-Chieh Huang, Lingzhe Zhang, Liancheng Fang, Xin Zou, Hanqian Li, Jiahao Huo, Yibo Yan, Zizhuang Deng, Lei Miao, Wei Guo, Haihong Tang, Bo Zheng, Philip S. Yu

llms agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Sep 22

Domain-decomposed Evolutional Deep Neural Network with Random Features for Transient Pressure Diffusion with Discontinuous and High-Contrast Coefficients

arXiv:2609.22349v1 Announce Type: cross Abstract: Transient pressure diffusion in heterogeneous porous media becomes difficult to resolve efficiently when permeability is discontinuous and spans seve...

By Peiqi Li, Jie Chen, Hui Zhang, Simon Hands

diffusion benchmarks

arXiv Machine Learning

Sep 22

Defusing Explosive Prompts: Understanding and Preventing Trigger-Based Prompt Injections in LLM Agents

arXiv:2609.22510v1 Announce Type: cross Abstract: As LLM applications integrate with external tools, they are increasingly exposed to indirect prompt injection (IPI), where adversarial instructions a...

By Justin Szczepaniak, Elad Feldman, Naum Viner, Ben Nassi

llms agents benchmarks safety

arXiv Machine Learning

Sep 22

Scalable Incremental Robustness Analysis of Neural Network Feedback Systems

arXiv:2609.22576v1 Announce Type: cross Abstract: Semidefinite programming (SDP) certificates for feedback systems containing deep neural networks (NNs) typically scale with the total number of neuro...

By Zichen Wang, Peter Seiler, Geir Dullerud, Bin Hu

benchmarks

Get the daily briefing