Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Computation and Language
Sep 24

Realize What Matters: Principled Context Representation for Large-Scale Reasoning

The paper introduces R3Con, a framework that applies principled design rules—derived from cognitive relevance theory—to construct effective context representations for large-scale reasoning tasks. By systematically organizing vast, heterogeneous information into manageable structures, R3Con outperforms nine state‑of‑the‑art baselines on two large‑document reasoning benchmarks, achieving up to 20 percentage points higher accuracy. Notably, R3Con enables smaller models (4B–9B) to surpass larger 35B models and even Claude Code, demonstrating that principled context representation can reduce dependence on model scale.

By Michael Theologitis, Dean Light, Shuyue Stella Li, Benjamin Newman, Yulia Tsvetkov, Dan Suciu
arXiv AI
Sep 24

PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety

PASTABench introduces a benchmark of 1,139 multi-turn trajectories to evaluate proactive safety monitoring in large language models. It formalizes three dimensions of intervention—whether, when, and what risk—to address gaps in step-level isolation and post-hoc trajectory assessment. The study finds that proactive intervention is largely unsolved, with the best model achieving only 40.74% optimal-timing interventions, and reveals that smaller models’ safety scores are often driven by lexical overfitting rather than true risk comprehension.

By Jiapeng Sun, Yujin Zhou, Han Zhu, Pengcheng Wen, Jiayi Zhou, Sirui Han, Yike Guo
arXiv AI
Sep 24

Do Center Biases Propagate? Robustness of Pathology Foundation Models in Whole-Slide Image Classification

The study investigates whether pathology foundation models (PFMs) carry center-related biases into whole-slide image (WSI) classification. By training models with increasing class-center correlations and evaluating six PFMs across four datasets and two MIL aggregators, the authors introduce the Area Under the Cramér's V Curve (AUCC) to measure both accuracy and degradation due to spurious correlations. Results reveal that center information propagates to WSI predictions, with robustness varying by PFM and MIL strategy, and that ComBat harmonization does not consistently improve robustness.

By Il\'an Carretero, Pablo Meseguer, Roc\'io del Amor, Valery Naranjo
arXiv AI
Sep 24

DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policies

DreamAvoid introduces a test‑time dreaming framework for Vision‑Language‑Action models to anticipate and avoid failures during critical manipulation phases. It uses a Dream Trigger to detect critical phases, samples candidate action chunks via an Action Proposer, and evaluates short‑horizon futures with a Dream Evaluator trained on success, failure, and boundary data. Experiments on real‑world and simulated tasks show DreamAvoid improves task success rates, achieving 72.5% success versus 48.8% for the base policy and 54.4% for GPC‑RANK.

By Xianzhe Fan, Yuxiang Lu, Shenyuan Gao, Xiaoyang Wu, Ruihua Han, Manling Li, Hengshuang Zhao
arXiv Machine Learning
Sep 24

Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streams

arXiv:2609. 27303v1 Announce Type: new Abstract: Livestreams are long-lasting interactive environments where audiovisual content, viewer activity, host behavior, and platform signals evolve together, creating assistance needs that emerge from the stream itself.

By Shujian Gao, Jiamei Yan, Yuchen Yang, Penghao Zhou, Qinglei Wang, Tiehan Fan, Yuan Wang, Zuxuan Wu, Yu-gang Jiang
arXiv AI
Sep 24

When Context Misleads: In-context Learning with Jurisdiction in Large Language Models

The paper "When Context Misleads: In-context Learning with Jurisdiction in Large Language Models" introduces FakeContextBench, a benchmark of pseudoscientific claims across seven domains to test large language models’ ability to judge whether contextual information should dictate the answer. Experiments show that both commercial and open‑source models struggle with context authority and that standard ICL fine‑tuning can worsen susceptibility to misleading context. The authors propose Jurisdiction In‑Context Learning (J‑ICL), a post‑training framework that adds context validation to the objective, which improves benchmark scores and reality accuracy by significant margins over baseline models.

By Pei-lin Li, Qingle Liu, Junyang Feng, Siyu Li, Sunqi Fan, Xin-Sheng Chen, Shuojin Yang
arXiv AI
Sep 24

Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning

The paper presents a benchmark to test whether vision‑language models can produce plant simulation configurations from images using in‑context learning. It focuses on cowpea plot reconstruction, requiring the models to output structured JSON that includes field and plant details. Open‑source multimodal models from the Gemma 4 and Qwen3.5 families are evaluated on synthetic and real drone datasets, using five in‑context methods, and the results show that while VLMs can generate valid JSON and estimate key agronomic metrics, their performance varies and often lags behind dataset baselines.

By Heesup Yun, Isaac Kazuo Uyehara, Earl Ranario, Lars Lundqvist, Christine H. Diepenbrock, Brian N. Bailey, J. Mason Earles
arXiv AI
Sep 24

MessyKitchens: Contact-rich object-level 3D scene reconstruction

MessyKitchens introduces a new dataset of cluttered real-world kitchen scenes with detailed 3D object shapes, poses, and accurate contact information. The authors extend the SAM 3D single-object reconstruction method with a Multi-Object Decoder (MOD) to jointly reconstruct entire scenes, achieving better registration accuracy and reduced inter-object penetration compared to prior work. The dataset, benchmark, code, and pretrained models will be publicly released on the project website.

By Junaid Ahmed Ansari, Ran Ding, Fabio Pizzati, Ivan Laptev
arXiv AI
Sep 24

Benchmarking Text-to-SQL under Role-Based Access Control

The paper introduces a new text‑to‑SQL benchmarking framework that incorporates realistic role‑based access control (RBAC) constraints. It augments existing benchmarks by generating plausible user roles and access policies through an LLM‑assisted workflow, followed by human‑in‑the‑loop quality control. The framework also provides evaluation metrics to detect RBAC‑specific failures and separate SQL utility from compliance, revealing that many high‑scoring models degrade sharply under access constraints.

By Yang Fei, Yangfan Jiang, Yin Yang, Xiaokui Xiao
arXiv AI
Sep 24

BIDETA: Brain-Inspired Data-Efficient Tactile Adaptation for Unseen Sensors

BIDETA is a gradient‑free framework that adapts pretrained tactile models to new sensors using only a few labeled target contacts. It preserves pretrained representations while repairing sensor‑dependent feature neighborhoods through rapid support memory, support‑conditioned spectral graphs, and reliability‑gated recurrence. Experiments on multiple datasets show that BIDETA dramatically improves accuracy and speeds up adaptation compared to prior methods.

By Boheng Liu, Lan Wei, Ziyu Li, Chenghua Duan, Qing Li, Dandan Zhang, Xia Wu
arXiv AI
Sep 24

World Models for Cross-Machine CNC Transfer under Partial Sensor Overlap

The paper investigates whether a command‑conditioned latent world model, trained on a source CNC machine with 17 sensor channels, can transfer its predictive capability to a target machine that shares only 10 of those channels. Experiments show that latent‑predictive pretraining offers no advantage over training from scratch on the source data, and that the transferred model outperforms a persistence baseline on the target but falls short of forecasters that normalize each input window by its own statistics. The study highlights that cross‑machine transfer under partial sensor overlap presents a unique challenge for command‑conditioned world models.

By Ayoub Louaye Bouaziz, Matthieu Ostertag, Anton Demasles
arXiv Machine Learning
Sep 24

The Drift Contract: Spectral Updates for Depth-Robust Local Learning

The paper introduces the Drift Contract, a spectral update geometry for local learning that improves depth robustness and hyperparameter stability. By applying momentum orthogonalization with spectral step scaling to per‑layer updates, the authors achieve consistent performance across a wide range of widths and depths on CIFAR‑10 MLPs, outperforming local Adam and providing a per‑layer, input‑conditioned drift bound. The study also shows that the spectral geometry itself, rather than step‑size rules, drives the observed depth robustness, while a negative result indicates that the stability benefit is limited to non‑normalized layers.

By Fabien Polly
arXiv Machine Learning
Sep 24

CORE-STACK+: Meta-Learning for Deep Stacked Generalization

CORE-STACK+ is a new meta‑learning framework for deep stacked generalization that tackles two key problems in heterogeneous vision ensembles: prediction‑space multicollinearity and calibration collapse. It introduces a four‑step preconditioning pipeline—kernelized redundancy filtering, a lightweight differentiable meta‑feature gate, a spectrum‑adaptive ridge penalty, and a Laplace‑approximate Bayesian blender—to jointly improve conditioning and calibration. Across six vision benchmarks, CORE‑STACK+ boosts accuracy, reduces model count and inference cost, and significantly lowers expected calibration error compared to existing methods.

By Noor Islam S. Mohammad
arXiv Machine Learning
Sep 24

When Post-Processing Fairness Constraints Help and When They Harm: Evidence from Eight Cross-Domain Evaluations

The paper introduces FAPE, a four‑stage framework for evaluating the post‑processing fairness intervention ThresholdOptimizer across eight diverse domains, including criminal justice, finance, healthcare, and education. It reports that the intervention reduces disparity in most high‑disparity cases but can worsen fairness when baseline disparities are low, and that a single deployment‑time audit is unreliable without continuous monitoring and baseline‑disparity screening.

By Nithin Raghava Ramachandra Narla
arXiv Machine Learning
Sep 24

Learning Risk Scores Robust to Unobserved Confounders

The paper introduces a method for learning risk scores that remain reliable even when historical data contain unobserved confounders. By treating propensity weights as uncertain and applying sensitivity analysis with Wasserstein distributionally robust optimization, the authors formulate a robust learning problem solvable via an exponential cone program. Experiments on semi‑synthetic UCI data show the approach improves calibration by up to 29.2% over traditional benchmarks and 11.1% over the state of the art, without harming other performance metrics.

By Ryan Edmonds, Yingxiao Ye, Sina Aghaei, Andr\'es G\'omez, \c{C}a\u{g}{\i}l Ko\c{c}yi\u{g}it, Phebe Vayanos
arXiv Machine Learning
Sep 24

ZO-COSMO: Index-Free One-Hop Mixing for Decentralized Zeroth-Order Optimization

The paper introduces ZO-COSMO, an index‑free one‑hop mixing scheme for decentralized zeroth‑order optimization that couples two‑query estimation with average‑preserving masked consensus. It characterizes the necessary one‑hop condition for sparse communication, derives sharp contraction bounds and convergence guarantees for core and sparse‑momentum updates, and demonstrates empirical gains on synthetic agents and Qwen LoRA workers, achieving notable accuracy improvements over traditional Rand‑k and all‑neighbor mixing methods.

By Shengjun Zhang, Tingyi Liu, Heng Zhang, Dong Xie
arXiv Machine Learning
Sep 24

Discover, Falsify, Revise: Auditing Input-Use Claims from Source Code to Predictive Contribution in Agent-Discovered Cell Models

The paper introduces CELLAUDIT, a method for auditing whether inputs claimed to influence predictive models actually do so. By testing if an input can enter the computation, whether predictions depend on it, and if that dependence improves observed responses, the authors evaluate agent-generated predictors on a morphology‑transcriptomics benchmark (BBBC047). Their findings show that many models claim compound contributions that are not supported by the data, and that falsification‑guided revisions can recover genuine input effects while improving performance.

By Mengran Li, Bo Li, Chengyang Zhang, Yang Yan, Jinfeng Xu, Zhenchao Tang