Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

Hugging Face Trending Papers
Sep 17

G^2RA-NET: Graph-based Cross-Slice Relation Modeling with Attention Gating for Medical Image Segmentation

G^2RA‑Net is a medical image segmentation framework that models cross‑slice relations using a graph‑based approach and an attention gating mechanism. It represents each image slice as a graph node, propagates semantic context through graph message passing, and then applies a Cross‑Slice Attention Gate to select relevant neighboring context and emphasize target anatomical regions. Experiments on brain MRI and abdominal CT datasets show that G^2RA‑Net improves segmentation accuracy and boundary quality compared to representative methods, with ablation studies confirming the effectiveness of its design.

Hugging Face Trending Papers
Sep 17

SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting

SETTer is a transformer-based model designed for long‑term multivariate time‑series forecasting. It introduces decoupled self‑attention and hybrid masking to better capture short‑ and long‑term patterns across time and channel dimensions, while adding simple explainable structures to highlight discriminative patterns. Experiments on real‑world benchmarks show that a single‑layer SETTer outperforms state‑of‑the‑art models in 88% of scenarios.

Hugging Face Trending Papers
Sep 17

FCA-Guided Counterfactual Explanations for Multi-Modal Breast Cancer Diagnosis: A Framework Achieving Perfect Validity with Emergent Sparsity

The paper introduces FCA‑Guided Counterfactual (FCA‑CF) explanations for multi‑modal breast cancer diagnosis, leveraging a Formal Concept Analysis lattice as a hard structural constraint to generate counterfactuals. On the TCGA‑BRCA dataset, FCA‑CF achieves perfect validity (100% prediction flips), the lowest average feature changes (2.37), and competitive proximity (0.900), outperforming four established counterfactual methods. Ablation studies show the lattice constraint and a greedy refinement phase are key to its sparsity and validity.

Hugging Face Trending Papers
Sep 17

Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence

The paper introduces the Profiling, Investigation, and Judgment (PIJ) benchmark, which contains 2,500 real homicide cases from five countries to evaluate large language models (LLMs) on pre‑arrest criminal investigation tasks. It assesses LLMs across criminal profiling, crime process reconstruction, and sentence prediction, revealing that performance drops as tasks require more implicit reasoning about unknown suspect profiles. The study finds that LLMs lag behind human experts, especially in inferential categories like motivation and victim‑offender relationships, and exhibit biases in gender, age, and motive attribution.

Hugging Face Trending Papers
Sep 17

Graph-Based Stochastic Power-UCT: Monte-Carlo Graph Search with Power Mean Estimation

The paper introduces Graph-Based Stochastic-Power-UCT (GS-Power-UCT), a Monte‑Carlo graph search algorithm that shares states reached at the same planning depth while maintaining separate values for different depths. It achieves an $O(n^{-1/2})$ convergence rate for the root estimate in finite‑horizon stochastic MDPs, matching tree‑based methods but with better sample reuse. Two full‑state variants, GS-Power-UCT‑F and GS-Power-UCT‑F$^+$, further explore sample sharing and bias control, with GS-Power-UCT‑F$^+$ converging to the optimal infinite‑horizon discounted value when the cross‑depth gap vanishes. Experiments on stochastic planning benchmarks demonstrate improved sample efficiency over existing tree‑based and graph‑based baselines.

Hugging Face Trending Papers
Sep 17

VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering

VākQA is a newly introduced benchmark for Telugu spoken factoid question answering, comprising 2,001 question‑answer pairs across six domains, 2.53 hours of speech audio, bilingual transcriptions, and human‑verified reference answers. The study validates automatic evaluation methods against human judgments, finding that Gemini‑as‑a‑judge best approximates human ratings but is inconsistently strict, while open‑weight judges tend to penalize correct Telugu answers that differ in surface form. Using this validated setup, the authors benchmark proprietary and open‑weight models, highlighting challenges such as cultural specificity loss in translation, phonetic confusions from speech input, and compounded errors from cascaded ASR‑MT pipelines.

Hugging Face Trending Papers
Sep 17

SlugTrails: An Egocentric Benchmark for Floor Plan Localization in Large Buildings

SlugTrails is a new egocentric benchmark for floor‑plan‑based indoor visual localization in large buildings, featuring 30 Hz Aria glasses recordings across three campus buildings and six floors (22 089 m²). The dataset includes CAD‑derived floor plans with semantic classes, circulation masks, and laser‑surveyed anchors, and supports three realistic sensing protocols: single walking frames, stationary multi‑view sweeps, and walking streams with odometry. Evaluation of five geometric and learned systems shows that stock models perform poorly, but fine‑tuning on SlugTrails significantly improves performance and cross‑dataset generalization, indicating that data scarcity limits current localization methods.

Hugging Face Trending Papers
Sep 17

Reproducing Transparent and Scrutable Recommendations: Exploring Open-Weight Models via Natural-Language User Profiles

The study reproduces a prior work on recommender systems that use generated natural‑language user profiles to enhance transparency and user control. It confirms that the User Profile Recommendation (UPR) model performs competitively and that altering these profiles uniformly shifts predicted ratings without changing ranking order. Additional experiments include context ablation, multi‑seed stability, and mechanistic interpretability analysis with the nnsight framework.

Hugging Face Trending Papers
Sep 17

Dictionary-Constrained Grapheme-to-Phoneme for Unsegmented Languages from LLM-Annotated Data

The paper introduces a context‑aware neural grapheme‑to‑phoneme (G2P) system for unsegmented languages like Japanese, combining a discriminative conditional random field over a dictionary‑based word lattice with large language model‑generated training data. By generating over two million synthetic sentences, the method addresses data scarcity and achieves superior performance compared to traditional morphological analyzers and neural sequence models. On the Joyo‑Kanji‑Yomi benchmark, it attains 99.62% target word reading accuracy, 0.32% target word phoneme error rate, and 0.14% sentence phoneme error rate.

Hugging Face Trending Papers
Sep 17

Rethinking Multi-Agent Collaboration: When More Is Less

The paper examines when multi‑agent collaboration is truly beneficial as large language models grow more capable. It finds that multi‑agent systems yield systematic advantages mainly for long‑horizon tasks with sparse dependencies, while single‑agent approaches excel in tightly coupled, sequential workflows. The authors introduce SAIGE, a lightweight, graph‑based collaboration framework that balances context efficiency and performance, demonstrating that adding more agents or deeper recursion does not always improve outcomes.

Hugging Face Trending Papers
Sep 17

MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration

MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration proposes a new quantization format that inverts the traditional microscaling approach by assigning private exponents to each element and a shared mantissa. The adaptive dual-format MiX-MX inference framework maps this format to a custom accelerator, replacing multipliers with shifters. Evaluations show that 4.5-bit MiX matches or surpasses NVFP4 accuracy on multimodal benchmarks while improving area efficiency by 25% and delivering 2.3–4.5× speedup with 1.4–2.9× energy reduction compared to the Focus accelerator.

arXiv Machine Learning
Sep 17

The Automaton Underneath: The Additive Input Pathway Is a Parasitic Attractor for State Tracking in Householder Linear RNN

arXiv:2609. 18966v1 Announce Type: new Abstract: Linear RNNs with input-dependent Householder-product transitions (DeltaNet/DeltaProduct-class) can provably represent hard state-tracking automata, yet trained models fail to length-generalize -- a gap recent work attributes to optimization, without a causal account.

By Gunner Levi Howe
arXiv Machine Learning
Sep 17

CompileRover: Revolutionizing Virtual Machine Compiler Optimization with a Tri-Role LLM-Driven Framework

CompileRover is a new optimization framework for virtual machine compilers that uses a tri‑role LLM‑driven collaboration mechanism involving a referee, an advisor, and an operator. It tackles common issues such as redundant computations, inefficient loops, and suboptimal function implementations by applying control‑flow analysis, code‑structure transformations, and dynamic execution pattern recognition. Benchmark evaluations show that CompileRover consistently outperforms existing virtual machine compilers, reducing execution overhead, improving data‑flow consistency, and enhancing overall compiler performance.

By Mingqiao Mo, Yunlong Tan, Hao Zhang
arXiv Machine Learning
Sep 17

Evidence-Grounded Agentic Formulation Development in an Autonomous Laboratory

The paper introduces Andromeda 2, an agentic system that uses structured in‑house experimental evidence to design and run successive batches of self‑emulsifying drug delivery systems (SEDDS). In a miniaturized automated lab, Andromeda 2 outperformed its predecessor Andromeda 1 and a traditional design‑of‑experiments campaign in developing paclitaxel formulations, achieving a 50 % hit rate versus 17 % and 2 % respectively, and identifying 12 formulations meeting all target product profile objectives. The system’s use of evidence‑grounded reasoning increased mean AUC by 34 % and produced a formulation with a 19 % w/w paclitaxel loading, roughly 3.3‑fold higher than a published benchmark.

By Michael M. Craig, Riley J. Hickman, Yingshan Ma, R\'emi Pich\'e-Taillefer, Christine Allen, Pauric Bannigan
arXiv Machine Learning
Sep 17

Modular Deep Learning Mechanisms for Auditable Next-Day Wildfire Spread Prediction

The paper presents modular deep learning augmentations for next‑day wildfire spread prediction, including wind‑ and slope‑conditioned attention biases, physics‑feature retrieval‑augmented output correction, and fire‑conditioned dual‑stream gating. These modules are evaluated on five backbone models using the Next Day Wildfire Spread benchmark, with staged ablations, directional audits, retrieval perturbations, calibration measures, and computational comparisons. The best augmented SwinUNETR model achieves an F1 score of 0.4216 and an AUC‑PR of 0.3673, while a mixed ensemble reaches 0.4292 and 0.3790, demonstrating that predictive performance, operational trustworthiness, and computational practicality can be simultaneously improved.

By Miguel Esparza, Aydin Ayanzadeh Ahmad Mousavi, Ali Mostafavi
arXiv AI
Sep 17

HPOQuest: A Rare-Disease Diagnostic Agent Using Active Phenotype Acquisition

HPOQuest is a training‑free framework that improves rare‑disease diagnosis by actively acquiring additional phenotypes. Starting from a few observed patient traits, it maintains a probabilistic ranking of possible diseases and iteratively selects follow‑up questions that are most informative. When clinicians confirm new phenotypes, the disease ranking is updated, and the set of candidate questions is refined, leading to significant gains—up to 30% in Recall@1 and 45% in Recall@5—across four benchmark cohorts.

By Kamilia Zaripova, Nassir Navab, Azade Farshad, Annalisa Marsico
arXiv AI
Sep 17

The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models

The paper examines the role of schema linking in Text-to-SQL systems and finds that recent large language models can effectively use relevant schema elements even when many irrelevant ones are present. Consequently, the authors eliminate schema linking when the entire schema fits within the model’s context window, instead employing augmentation, selection, and correction techniques to enhance accuracy. Their approach achieves first place on the BIRD benchmark with a 71.83% accuracy.

By Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, Amine Mhedhbi
arXiv AI
Sep 17

QiT: Quantum-Inspired Transformer for Visual Recognition Task

QiT is a Quantum‑Inspired Transformer designed for visual recognition tasks. It replaces quantum neural network concepts with scalable classical operations: angle‑inspired encoding of image tokens, self‑attention over periodic features approximating quantum fidelity kernels, and gated multiplicative emulation of variational circuit interactions. The model achieves competitive performance on image‑classification benchmarks, matching a classical Transformer while avoiding the high runtime costs of simulated quantum models.

By Badri N. Patro, Vijay Agneeswaran
arXiv AI
Sep 17

Learning Nuclear Structure with AI: Radii and Collectivity

The paper presents NuCLR, a multi-task neural network that learns nuclear data representations to predict charge radii and electric‑quadrupole transition strengths across hundreds of nuclides. Using held‑out ensembles, the model achieves a charge‑radius RMS deviation of 0.0147 fm and a B(E2) RMS deviation of 0.192 e²b², comparable to leading nuclear models. The authors provide error bars indicating where additional experimental data could improve predictions, positioning NuCLR as a data‑driven surveyor of nuclear structure.

By Giuliano Giacalone, Sokratis Trifinopoulos, Mike Williams
arXiv AI
Sep 17

${M}^2$Tok: Multi-head Multi-codebook Discrete Action Tokenization for Vision-Language-Action Models

The paper introduces ${M}^2$Tok, a Multi-head Multi-codebook Action Tokenizer that reduces reconstruction loss in discrete action tokenization for Vision‑Language‑Action models. By decomposing latent action features into multiple heads and assigning independent codebooks to each, the tokenizer expands representational expressivity and improves policy performance. Experiments on RoboTwin, Simpler‑Env, and zero‑shot real‑world tasks show superior reconstruction fidelity and higher success rates compared to prior methods.

By Chunpu Xu, Zhixuan Liang, Yuhao Zhang, Chi-Min Chan, Jessie Wang, Yang Xiao, Mengkang Hu, Xiaokang Yang, Yao Mu