Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

Hugging Face Trending Papers
Oct 28

Fed-IGTD: A federated learning-based framework for tabular-to-image encoding

Fed-IGTD is a federated learning framework that enables tabular-to-image encoding without centralizing data. It reconstructs a global correlation matrix from local statistics and trains a CNN via Federated Averaging, while applying two differential privacy mechanisms: a one-shot Gaussian noise addition during correlation aggregation and a client‑level DP‑FedAvg during training. Experiments on five datasets show that privacy protection during mapping construction has minimal impact on performance, whereas DP‑FedAvg can significantly degrade utility, especially with small client cohorts and moderate privacy budgets.

Hugging Face Trending Papers
Oct 28

What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges

The paper introduces MIMO-Bench, a new benchmark designed to test Multimodal Large Language Models (MLLMs) on visual reasoning tasks that involve multiple instances of the same object category. Unlike existing benchmarks that focus on images with few object instances, MIMO-Bench contains 3,802 image-question pairs requiring the model to disambiguate among similar objects using attribute binding and numerical counting. Results show that state‑of‑the‑art models, both open‑source and proprietary, perform significantly worse on this benchmark, underscoring their limitations in handling multi‑instance, multi‑object scenarios.

Hugging Face Trending Papers
Oct 28

How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0

The study compares deterministic static application security testing (SAST) with AI coding agents and AI security harnesses using the VulnBench 2.0 benchmark of 108 vulnerabilities across multiple languages. Nine productized configurations were evaluated, with Snyk Code achieving 82.4% macro recall and Claude Code at 79.3%; together they covered 101 of 108 vulnerabilities, exceeding any single configuration. The results show that AI engines’ effectiveness depends heavily on their harness policies and that AI tools can uncover real vulnerabilities absent from the reference set.

Hugging Face Trending Papers
Oct 28

Holistic Diversity Management in LLM-Guided Evolutionary Search

The paper introduces HoliDivEvolve, a holistic strategy for maintaining diversity in LLM-guided evolutionary search. It employs diverse initialization, trajectory‑adaptive island selection, and growth‑adaptive migration to ensure distinct search directions and effective knowledge sharing across islands. Experiments on varied benchmarks demonstrate competitive performance and provide controlled analyses of the design choices.

arXiv Machine Learning
Oct 28

Fully quantum training of parametric circuits with Quantum Hamiltonian Descent

The paper introduces a fully quantum method, Quantum Hamiltonian Descent (QHD), for training parametric quantum circuits by treating circuit parameters as quantum particle positions in a potential defined by the cost function. It demonstrates that, for smooth periodic cost functions, the simulation error is bounded independently of grid resolution and grows at most linearly with the number of parameters, while the dynamics depend on a single schedule curve that can be optimized to reduce gate count. Numerical simulations on benchmark functions and circuits show that schedules designed in this way significantly improve the probability of locating the global minimum compared to conventional approaches.

By Marco Burato, Leonardo Banchi
arXiv Machine Learning
Oct 28

Holo-NN: Holographic Nearest Neighbor Search via Compressive Sensing

Holo‑NN introduces a new algebraic framework for privacy‑preserving approximate nearest neighbor search that replaces traditional tree or graph indexing with a single linear projection. By exploiting the sparsity of neighborhood membership and combining locality‑preserving transforms with compressive sensing, the entire database is folded offline into a compact holographic sensing matrix. At query time, encrypted search reduces to a single, non‑interactive matrix‑vector multiplication, eliminating the need for FHE bootstrapping, multi‑round interaction, or client‑side index state, and enabling efficient, accuracy‑tunable retrieval on CPUs and GPUs.

By Bryant Gipson
arXiv Machine Learning
Oct 28

DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving

DualPace is an LLM serving runtime that decouples scheduling between a GPU engine and an AMX‑enabled CPU engine, allowing the CPU to independently batch and advance decodes while still cooperating with the GPU for dense stages. This design expands key‑value cache capacity into host memory and reduces bursty CPU progress, leading to significant throughput gains—up to 43.2 % over NEO and 90.6 % over vLLM—without increasing latency. The system also maintains low time‑to‑first‑token and supports higher request rates at comparable tail latency.

By YuXuan Lin, KeQiang Wu, Kingsum Chow
arXiv Machine Learning
Oct 28

EmoNet: A Lightweight Attention-Based CNN for Speech Emotion Recognition

EmoNet is a lightweight attention‑based CNN framework designed for speech emotion recognition across seven categories. It combines a four‑block depthwise separable CNN encoder with squeeze‑and‑excitation attention, residual connections, a Bidirectional LSTM, and multi‑head attention pooling, using a 134‑channel acoustic feature set and data augmentation to enhance robustness. On a combined TESS, RAVDESS, and SAVEE dataset, EmoNet achieves 96.75% test accuracy with only 4.11 million parameters, outperforming a reproduced baseline and competing with larger state‑of‑the‑art models.

By Muhammad Moiz Khalid, Abdur Rehman Tariq, Muhammad Okasha Yamin
arXiv Machine Learning
Oct 28

TCSP3.0: Template-Based Crystal Structure Prediction with Coefficient-Preserving Element Substitution and Structural Diversity

TCSP3.0 is an improved template‑based crystal structure prediction method that enforces coefficient‑preserving element mappings, substitutes species simultaneously on parsed structures, and validates each candidate’s composition and cell. It adds periodic‑number retrieval, soft oxidation‑state evidence, radius‑compatibility scoring, structural deduplication, and ORB‑v3 machine‑learning‑potential relaxation and energy ranking, achieving Top‑1 and Top‑20 match rates of 72.2 % and 90.6 % on the T180 benchmark and outperforming previous TCSP2.0 and several deep generative models. The method runs in a median of 28 s per target on two CPU cores, roughly ten times faster than CrystalFlow, with most gains attributed to the correctness fixes and structural deduplication.

By Ying Feng, Jeffrey Hu, Lai Wei, Liang Tao, Madeline Miklos, Dipannoy Das Gupta, Hossein Erfanshekooh, Jianjun Hu
arXiv Machine Learning
Oct 28

AudioFM: Towards General Purpose Audio Representation Learning

AudioFM is a self‑supervised framework that learns audio representations by combining recording‑level contrastive learning with patch‑wise masked latent discrimination. The model uses visible acoustic context for a student to predict masked patch representations and matches them to full‑view teacher targets from other recordings, while also incorporating weaker supervision from temporal neighbors to promote local consistency. Evaluations show that AudioFM achieves state‑of‑the‑art results across diverse audio domains in linear probing, fine‑tuning, and zero‑shot transfer, excelling in event detection, scene‑level understanding, and domain generalizability.

By Yamin Li, Ruidong Zhang, Anran Wang, Eduardo Fonseca, Dan Ellis, Zechen Li, Simon A. Lee, Tao Pang, Jake Garrison, Maxwell A. Xu, Yicheng Wang, Jitian Zhang, Weizhi Zhang, Haichao Zhang, Catie Chang, Xin Liu, Anupam Pathak, Daniel McDuff, Shwetak Patel, Jake Sunshine, Yuzhe Yang, Xiaoran Fan
arXiv Machine Learning
Oct 28

Intellectual Property Protection of Hyperdimensional Computing Classifiers: Black-Box Ownership Verification via Model Watermarking

The paper investigates model watermarking for hyperdimensional computing (HDC) classifiers, adapting backdoor-based embedding to three prototype‑learning mechanisms. Across four benchmarks, error‑driven models (AdaptHD and OnlineHD) support ownership verification with minimal accuracy loss, while single‑pass Vanilla HDC requires higher trigger exposure and incurs larger accuracy penalties. The study also shows that watermarks can be overwritten, survive certain stress tests on FPGA, but may be lost during extraction or severe stress, highlighting both feasibility and limitations of HDC watermarking.

By Hamed Poursiami, Farshad Soleiman, Ahmad Tahmasivand, Guojing Cong, Thomas Potok, Khaled N. Khasawneh, Kai Zeng, Maryam Parsa
arXiv Machine Learning
Oct 28

One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference

The paper investigates how to optimally partition streaming inference models into megakernels and conventional graph components. It introduces a framework that predicts the best megakernel composition without full end‑to‑end runs, showing that multiple megakernels or a fully conventional graph often outperform a single megakernel across various models and batch sizes. The study defines four model categories based on megakernel stability and demonstrates significant latency and throughput gains on an H100 NVL GPU using a three‑megakernel design.

By Aditya Chatterjee, Anand Joseph X M, Sanjay Singh Chauhan
arXiv Machine Learning
Oct 28

FeatureLoop: Benchmarking Coding Agents under Evolving Requirements

FeatureLoop is a benchmark that evaluates coding agents on 61 human‑reviewed tasks drawn from 34 repositories in five programming languages. Each task tracks a development goal through iterative feedback—adding or clarifying requirements and reporting failures—while cumulative executable checks monitor new behavior, repairs, and preservation of earlier functionality. In tests of 12 state‑of‑the‑art models, the best agent completed 72.1% of tasks with both feedback types, but only 15.4% of failures were recovered within two revisions, highlighting issues such as incomplete repairs, regressions, and token inefficiency.

By Jia Feng, Alyssa Wu, Cuiyun Gao, Yunlong Wang, Songyuanyi Lu, Zhichao Hu
arXiv Machine Learning
Oct 28

TRON: Transferring Naturally Evolved Network Architectures into Neuroevolution

TRON (TRansfer Of Natural networks) is a neuroevolution framework that uses experimentally characterized protein‑protein interaction (PPI) networks as architectural priors instead of random topologies. By combining classical evolutionary operators with neuroevolution techniques such as speciation and species quarantine, TRON improves the preservation of functional organization during mutation and recombination. Evaluations on six benchmark datasets show that TRON achieves the best predictive performance on all classification tasks and most regression tasks, with lower variability across runs, indicating greater robustness and convergence stability at the cost of higher computational effort.

By Konstantinos Zacharopoulos, Alexandra Bekou, Jose M Serradell, Foteini Paraskevopoulou, Apostolos Fournaris, Ioannis Arapakis, Oscar Lao
arXiv Machine Learning
Oct 28

Unified Robust and Convergent Over-The-Air Federated Learning under Channel Noise and Heterogeneity

The paper introduces RoCo-FL, a robust and convergent framework for Over-the-Air Federated Learning that addresses channel noise, fading, and data/system heterogeneity. RoCo-FL models local updates with inexact optimization and a proximal constraint to stabilize updates under wireless impairments and stragglers. The authors provide convergence guarantees for non-convex objectives and demonstrate improved stability and accuracy on CIFAR10, CIFAR100, and FEMNIST compared to existing baselines.

By Zubair Shaban, Nazreen Shah, Ranjitha Prasad
arXiv Machine Learning
Oct 28

GateTS: Versatile and Efficient Forecasting via Attention-Inspired routed Mixture-of-Experts

GateTS introduces a streamlined univariate forecasting model that merges sparse Mixture-of-Experts computation with an attention-inspired gating mechanism, eliminating the need for auxiliary load‑balancing losses. The architecture achieves superior predictive accuracy across both short‑ and long‑term horizons, including intermittent signals, while using far fewer parameters than leading transformer models such as PatchTST. Empirical tests on diverse datasets show GateTS is computationally more efficient than LSTM, enabling cost‑effective inference for real‑world time‑series applications.

By Kyrylo Yemets, Mykola Lukashchuk, Ivan Izonin
arXiv Machine Learning
Oct 28

Coder as Editor: Code-driven Interpretable Molecular Editing

The paper introduces MECo, a framework that translates molecular editing intentions into executable code, bridging the gap between reasoning and execution in drug discovery. MECo generates human‑interpretable editing intentions from a molecule and design objective, then converts those intentions into precise structural edits. The method achieves over 98% accuracy on realistic edits and improves downstream benchmark consistency by 38–86 percentage points, outperforming SMILES‑based baselines while maintaining structural similarity.

By Wenyu Zhu, Chengzhu Li, Xiaohe Tian, Yifan Wang, Yinjun Jia, Jianhui Wang, Bowen Gao, Haichuan Tan, Ya-Qin Zhang, Wei-Ying Ma, Yanyan Lan