Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning

Oct 15

Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization

arXiv:2610.19084v1 Announce Type: new Abstract: Automatic prompt optimization (APO) uses feedback from model responses to improve prompts automatically. However, its effectiveness is usually measured...

By Chinmoy Mitra, Tarek Mahmud

llms benchmarks safety

arXiv Machine Learning

Oct 15

FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing

arXiv:2610.19240v1 Announce Type: new Abstract: Financial disclosures express the underlying relationships of double-entry accounting through unstructured text. While financial LLM benchmarks predomi...

By Ashman Mehra, Bryan Routledge, Jane JaeYeon Pyo, Pierre Jinghong Liang

llms nlp benchmarks safety

arXiv Machine Learning

Oct 15

Why don't Graph Transformers attend to the same token?

arXiv:2610.19324v1 Announce Type: new Abstract: In large language models (LLMs), attention concentrates heavily on the first token. This 'attention sink' is believed to protect deep models against ov...

By Alexandre Bismuth

llms benchmarks

arXiv Machine Learning

Oct 15

W-JEPA: Wavelet Joint-Embedding Predictive Architecture for Wearable Activity Recognition

arXiv:2610.19411v1 Announce Type: new Abstract: A major challenge in wearable device activity recognition is the heterogeneity across datasets in sensor placement, channel count, sampling rate and su...

By Yogesh Prabhu, Loki Natarajan, Jingjing Zou

rag nlp benchmarks

arXiv Machine Learning

Oct 15

Progress-Aware Distillation for Mitigating Stagnation in Small Language Model Agents

arXiv:2610.19439v1 Announce Type: new Abstract: While LLM-based agent systems have demonstrated remarkable proficiency in reasoning and solving complex real-world tasks, they also come with substanti...

By Yuanpu Cao, Saket Sathe, Hanyu Wang, Ziyi Yin, Fenglong Ma, Jinghui Chen

llms agents fine-tuning efficiency benchmarks

arXiv Machine Learning

Oct 15

Function-Space Temporal Self-Distillation for Policy Optimization

arXiv:2610.19687v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over respon...

By Aryan Das, Arnab Bhattacharya, Vinay Kumar Verma

llms reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Oct 15

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

arXiv:2610.19700v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, bu...

By Fanyao Li, Yuxi Liu, Xiaohao Liu, He Chang, Zhaoxin Wang, Zhulin Tao

fine-tuning benchmarks

arXiv Machine Learning

Oct 15

Reinforcement Learning via Advantage Guidance

arXiv:2610.19783v1 Announce Type: new Abstract: Reinforcement learning (RL) holds the promise of learning a policy that can outperform what is shown in the data. However, in practice, such systems su...

By Zhiyuan Zhou, Sergey Levine

diffusion reinforcement-learning benchmarks

arXiv Machine Learning

Oct 15

HGRP: Heterogeneous Graph Neural Network for Routability Prediction

arXiv:2610.20209v1 Announce Type: new Abstract: Routability prediction enables rapid evaluation of placement quality without routing, providing effective feedback for routability-driven placement opt...

By Yule Sheng, Yue Yuan, Shan Shen, Daying Sun

benchmarks

arXiv Machine Learning

Oct 15

MAEM: Mitigating Recency Bias via a Replay-Free, Capacity-Controlled Dynamic Mixture of Experts for Class-Incremental Graph Learning

arXiv:2610.20238v1 Announce Type: new Abstract: Graph Neural Networks (GNNs) excel on graph-structured data but suffer catastrophic forgetting when classes arrive sequentially. In class-incremental g...

By Junlin Chen, Wenhao Wang, Junchao Xu, Qingsong Yao, Fei Xue, Li Sun, Xiao Wang, Ruijie Wang

fine-tuning benchmarks safety

arXiv Machine Learning

Oct 15

FUSE: Fairness-aware Update Screening against Byzantine Exploitation in $q$-Fair Federated Learning

arXiv:2610.20264v1 Announce Type: new Abstract: Client-level fair federated learning, in the form of $q$-fair federated learning ($q$-FFL), agnostic federated learning, or tilted empirical risk minim...

By Tiantong Sun, Jinlin Liu, Xingru Wei, Zequan Fang, Wei Chen, Qiang Yang, Xiaojin Zhang

benchmarks safety

arXiv Machine Learning

Oct 15

SpectrumQuant: Spectrum Alignment for LLM Quantization

arXiv:2610.20303v1 Announce Type: new Abstract: Orthogonal transforms have emerged as an effective tool for low-bit quantization of large language models (LLMs), as they reshape activation distributi...

By Dwaipayan Haldar, Indranil Patra, Azhar Yousuf, Chandra Sekhar Seelamantula

llms efficiency benchmarks safety

arXiv Machine Learning

Oct 15

Fractional is Better: Learnable Derivative Orders in Neural Operator Learning

arXiv:2610.20363v1 Announce Type: new Abstract: Neural operators learn mappings between function spaces, enabling fast surrogate solutions to partial differential equations. Despite remarkable archit...

By Fares B. Mehouachi, Saif Eddin Jabari

benchmarks safety

arXiv Machine Learning

Oct 15

Detection of Potato Plant Leaf Diseases using Convolutional Neural Networks

arXiv:2610.20544v1 Announce Type: new Abstract: Potato is an important food-security crop in Ethiopia, but its production is affected by foliar diseases, particularly early blight and late blight. Th...

By Haimanot Bitew Alemayehu, Simachew Melaku Ayalew, Chilotie Desalew Mengesha, Ayisheshim Almaw Bogale, Anteneh Birhan Yenew, Abrham Tegegne Bogale

benchmarks

arXiv Machine Learning

Oct 15

Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

arXiv:2610.20572v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reac...

By Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes, Weitong Zhang, Taylor W. Killian

efficiency benchmarks

arXiv Machine Learning

Oct 15

Bayesian Causal Forests for Zero-Inflated Continuous Outcomes

arXiv:2610.18889v1 Announce Type: cross Abstract: Estimating (heterogeneous) treatment effects is still challenging when the outcome is semicontinuous (i.e., exhibiting both a discrete point mass at...

By Hugo Gobato Souto, Eric Wang, Igor Shuryak

benchmarks safety

arXiv Machine Learning

Oct 15

Distinguishing Process Change, Instrument Fault and Data-Path Corruption in Streaming Telemetry: A Reference Implementation and Benchmark

arXiv:2610.18904v1 Announce Type: cross Abstract: A reading departs from what a monitoring system expects for one of three reasons: the physical process changed, the instrument is failing, or the val...

By Oleksandr Klymenko

benchmarks

arXiv Machine Learning

Oct 15

TorchSpin: Differentiable Electron Paramagnetic Resonance Simulation in PyTorch

arXiv:2610.18924v1 Announce Type: cross Abstract: Interpreting electron paramagnetic resonance (EPR) spectra commonly requires repeated forward simulation and optimization of spin Hamiltonian paramet...

By Alec H. Follmer

benchmarks

arXiv Machine Learning

Oct 15

The Agentic Scientist: Orchestrating Coding Agents for Automated Scientific (Co-)Discovery

arXiv:2610.18933v1 Announce Type: cross Abstract: Modern coding agents are fundamentally reshaping the role of human input for computational research: they write code, launch in silico experiments, a...

By Robert Tjarko Lange, Luca Grillotti, Yutaro Yamada, Rujikorn Charakorn, David Ha

agents benchmarks

arXiv Machine Learning

Oct 15

Physical Layer Anomaly Detection via Adaptive Gaussian Mixture Model

arXiv:2610.18946v1 Announce Type: cross Abstract: Physical-layer spoofing attacks pose a critical threat to wireless localization systems by manipulating channel state information (CSI). Detecting su...

By Mamady Delamou, Luan Chen, Arsenia Chorti

benchmarks

Get the daily briefing