Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

Hugging Face Trending Papers
Oct 8

The Lattice of Transition Laws

The paper investigates how to predict the performance of decoding schedules for generative models that blend diffusion and autoregressive (AR) approaches. By modeling diffusion, AR, and hybrid models as paths on a corruption lattice, the authors define a schedule cost based on discarded parallel steps and show that the minimal zero‑cost schedule is determined by the data’s geometry, such as the treedepth of a Markov graph. They further demonstrate that schedules with fewer steps than the treedepth incur a positive cost, which can be ranked in advance using a kernel derived from pretrained weights, and validate these predictions across text, image, and video generation tasks.

arXiv Computer Vision
Oct 8

CIRRA: Dual-Level Continual Instruction Reconciliation with Ongoing Execution for Embodied Robot Agents in Interactive Household Tasks

arXiv:2610.08862v1 Announce Type: cross Abstract: Household robots must accommodate new user instructions while executing ongoing tasks. Existing agents often regenerate or extensively revise the rem...

By Ci Zhang, Enfu Nan, Arman Akbari, Lin Zhao, Li Wang, Chen Wang, Weiwei Chen, Yanzhi Wang, Geng Yuan
arXiv Computation and Language
Oct 8

BEACON-SP: Ontology-Grounded GraphRAG Framework for Clinical Suicide Risk Assessment

BEACON-SP is an ontology‑grounded Graph Retrieval‑Augmented Generation framework designed to aid clinicians in suicide risk assessment by integrating diverse clinical, behavioral, social, and temporal data. It constructs patient knowledge graphs guided by a unified suicide prevention ontology that merges the Three‑Step Theory, the Integrated Motivational‑Volitional Model, and the Suicide Social Determinants of Health Ontology. In a benchmark of 1,500 queries across 15 clinical categories, BEACON‑SP outperformed a vector‑based RAG baseline in completeness, clinical relevance, and evidence grounding, with 76.4% of paired comparisons favoring the GraphRAG approach.

By Kemal Davaslioglu, Nathan Conger, Sastry Kompella, Yalin E. Sagduyu, Nathaniel D. Bastian
arXiv Machine Learning
Oct 8

MORA: Modeling Observed Changes for Drift-Robust Time-Series Anomaly Detection

MORA is a drift‑robust time‑series anomaly detection framework that addresses the ambiguity between distribution drift and true anomalies by defining a temporal change disambiguation problem. It reconstructs a local target from paired short‑ and long‑term views, using the reconstruction gap as a measure of contextual support to adjust the primary anomaly score. The method requires no drift annotations or online adaptation and demonstrates strong robustness to non‑stationarity while maintaining sensitivity to genuine anomalies on four TSAD benchmarks.

By Xudong Mou, Tiejun Wang, Rui Wang, Hui Wang, Pin Liu, Tianyu Wo, Xudong Liu, Renyu Yang
arXiv Computer Vision
Oct 8

How Private is Private? A Comparative Study for Face De-Identification

The paper introduces a unified evaluation framework for face de-identification (FDeID), addressing inconsistencies in current protocols. It presents UtilFace, a demographically balanced benchmark assembled from four large-scale datasets, and HiFD, a hierarchical metric that simultaneously measures identity suppression, multi-level utility preservation, and image quality. Using this framework, the authors compare adversarial, GAN-based, and diffusion-based methods, revealing trade-offs and failure modes that were previously hidden, and they release the benchmark and toolkit for reproducible research.

By Hui Wei, Hao Yu, Hui Kuurila-Zhang, Guoying Zhao
arXiv Computation and Language
Oct 8

The Long Road to the Same Answer: Cognitive Bias Under Escalating Reasoning Budgets in Large Language Models

The study investigates whether large language models (LLMs) that allocate extra computation during inference—termed reasoning models—reduce classic decision biases compared to their non‑reasoning counterparts. Using 30 vignettes covering six cognitive biases and varying token budgets up to 8,192 tokens, the authors find that reasoning models are not less biased, and increased deliberation does not reliably diminish bias magnitude. Only anchoring showed a bias in the human direction, while other biases either remained unchanged or moved further from human patterns, suggesting that test‑time reasoning does not guarantee rationality.

By Obada Kraishan
arXiv Computation and Language
Oct 8

When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs

The paper introduces TriviaRoomQA, a multilingual benchmark of 3,300 parallel multiple-choice questions across 288 topics, plus an additional 5,340 French-only questions. It evaluates 30 open-weight large language models (7–70 B parameters) from various regions, finding strong performance on knowledge-intensive subjects like history and geography but weaker results on everyday popular‑culture topics such as celebrities, music, movies, and news. The study also shows that model performance varies across languages for the same questions, indicating that factual knowledge access is not always language‑independent.

By Anna Mosolova, Djam\'e Seddah
arXiv Computer Vision
Oct 8

WAPR: A Foundation Model for Wide-Angle Refinement in Unseen Object Pose Estimation

WAPR is a zero‑shot wide‑angle pose refinement model that can correct candidate 6D poses with rotational errors up to 90°, achieving fast inference (≤1 s per frame) and high throughput (≈25 detections per second). It leverages rotational symmetry priors to canonicalize pose targets and introduces the SA6D dataset, which augments 944 GSO scans into ~50 k object instances and ~2 M RGB‑D images. Experiments on seven BOP core datasets demonstrate that WAPR sets new state‑of‑the‑art performance for unseen‑object pose estimation in both fast and unconstrained settings.

By Yulin Wang, Mengting Hu, Hongli Li, Jianghao Zhou, Chen Luo
arXiv Computer Vision
Oct 8

MeshCarve: Artisan Mesh Generation with Flow Matching in Compact Latent Spaces

MeshCarve introduces a flow‑matching approach that generates artisan meshes entirely within compact latent spaces, separating vertex positions and edge connections to avoid joint encoding issues. It employs a hierarchical sparse transformer backbone with VertexVAE and EdgeVAE, which anchor on discrete vertices to drastically shorten token sequences and use spatial‑aware compression without sacrificing reconstruction quality. The method outperforms state‑of‑the‑art autoregressive and flow‑matching techniques on Objaverse and generalizes to Toys4K, representing one of the first fully compressed latent artisan mesh generators.

By Xiyu Wang, Ruocheng Wu, Yufei Wang, Zhihao Li, Lanqing Guo, Bihan Wen
arXiv Machine Learning
Oct 8

Domain-informed Adaptive Sampling for Generalizable PINNs in Metal Additive Manufacturing via Conditional Flow Matching

The paper introduces a domain-informed adaptive sampling method for physics-informed neural networks (PINNs) applied to metal additive manufacturing. By combining a conditional Flow Matching model with a domain-informed base distribution, the approach generates adaptive collocation points that better capture process-dependent heat dissipation regions. Experiments on numerical benchmarks show a 62.1% reduction in relative L2 error compared to state‑of‑the‑art PINN baselines.

By Hyeonsu Lee, Jihoon Jeong