Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning
Oct 19

LagNet: Lag-Structured Operators for Long-Term Time Series Forecasting

LagNet introduces LagLayer, a Toeplitz‑structured operator that ties history‑to‑future connections by relative lag, replacing traditional temporal linear layers in forecasting models. By stacking LagLayers with activations and multiple feature maps, LagNet achieves competitive performance on long‑term time series benchmarks. Sensitivity matrix diagnostics and limited‑data experiments show that LagLayer offers a sample‑efficient alternative to linear layers.

By Zhenan Yu, Guangxin Jiang, Jiuyun Jiang, Jin Yang
arXiv Machine Learning
Oct 19

CocoRNA: Collective RNA Design with Cooperative Multi-agent Reinforcement Learning

CocoRNA is a cooperative multi-agent reinforcement learning framework that tackles RNA inverse design by breaking the problem into smaller sub‑problems solved collaboratively by multiple agents. A centralized critic provides global structural guidance during training, allowing the agents to jointly optimize their design strategies. Experiments on the Rfam dataset show that CocoRNA outperforms state‑of‑the‑art methods in both success rate and design speed, and additional tests on other biological sequence design tasks demonstrate its broad effectiveness.

By Tianmeng Hu, Biao Luo, Ke Li
arXiv Machine Learning
Oct 19

Has GPT-6-Astra Achieved Embodied Intelligence?

The paper evaluates GPT‑6‑Astra’s performance across 12 robotic benchmarks, showing it achieves unprecedented results when using in‑context learning from training‑set demonstrations. It highlights the benefits and costs of a dual‑system architecture (high‑level reasoning and low‑level control) and identifies practical gaps in real‑world deployment, such as expensive demonstration collection and physical constraints. The authors propose a new benchmark prototype that derives tasks from real‑world needs, omits in‑domain demos, and measures agents against a continuously running world clock.

By SenseNova, Joint Research Team, :, Zhongang Cai, Ziang Cao, Tailai Chen, Han Chong Chua, Linjun Dai, Hanming Deng, Hokin Deng, Haiwen Diao, Chenjing Ding, Weichen Fan, Xiao Fu, Zerui Gong, Chenyang Gu, Zexin He, Zhi Hou, Bo Li, Qinyue Li, Quanzhou Li, Xiaowan Li, Zhichao Liao, Dahua Lin, Zilong Lin, Bo Liu, Chunxiao Liu, Pengfei Liu, Tianqi Liu, Ziwei Liu, Chen Change Loy, Lewei Lu, Jiapeng Luo, Yuexiao Ma, Liang Pan, Xingang Pan, Oscar Qian, Ziyi Shang, Yuyao Shao, Dawei Su, Dacheng Tao, Qingyi Tao, Quan Wang, Ruisi Wang, Xiaogang Wang, Ziyi Wang, Beichen Wen, Donghao Wu, Jiaxu Wu, Silei Wu, Xiaoming Wu, Zhonghua Wu, Hongcan Xiao, Weiye Xu, Atsushi Yamashita, Lei Yang, Zhe Yang, Baiqiao Yin, Wanqi Yin, Hanfei Zhao, Tianbo Zhong, Long Zhuo, Jing Zuo
arXiv Machine Learning
Oct 19

Towards Reconstructing Geographically Diverse Architecture with 3D Foundation Models

The paper introduces ArchWorld, a benchmark of 3D architectural landmarks with extensive geographic coverage and detailed metadata, to evaluate and analyze the performance of 3D foundation models (3DFMs). Through a comprehensive error analysis, the authors uncover geographic disparities in model accuracy that stem from variations in data representation, scene size, and region-specific architectural features. They propose an intervention scheme that enhances 3DFM performance while simultaneously reducing these geographic disparities.

By Nicky Kriplani, Yiwen Zhang, Angelina Wang, Hadar Averbuch-Elor
arXiv Machine Learning
Oct 19

Shift-Reliance Alignment: Ranking Vision-Language Models by the Cues They Trust

The paper introduces Shift‑Reliance Alignment (SRA), a method for ranking vision‑language models (VLMs) based on how much they rely on specific cues such as scene context or text. By editing source images to mislead these cues and measuring model responses, SRA predicts which VLMs will be misled by physical sticker attacks and which will hallucinate on unedited photographs. In pre‑registered tests, SRA selects models that reduce error on challenging datasets like Objects365 and OpenImages, outperforming selection by source error and model confidence on the CueDial benchmark.

By Faisal Mohammad, Mohamed Younis
arXiv Machine Learning
Oct 19

CoLU: Confidence-Weighted Layerwise Unlearning for Large Language Models

The paper introduces CoLU, a confidence‑weighted layerwise unlearning method for large language models that updates only a single transformer layer instead of the entire model. It also proposes Brief Layer Screening (BLS) to quickly rank layers using a normalized forgetting‑retention score, reducing CoLU’s runtime by 60–80% while maintaining performance. Experiments on TOFU, MUSE Books, and News datasets show CoLU outperforms state‑of‑the‑art methods in both forgetting and retention, and uses significantly less GPU memory.

By Jingkai Guo, Ziyi Zhao, Chaitali Chakrabarti, Deliang Fan
arXiv Machine Learning
Oct 19

Controlling the Diversity-Quality Trade-Off of LLMs by Merging Models Before and After Alignment

The paper proposes a depth‑aware FrankenMerging technique that blends lower layers of an aligned large language model (LLM) with higher layers of its base counterpart to balance diversity and quality without extra training. Experiments demonstrate that adjusting the layer split controls both lexical and semantic diversity, often surpassing the aligned model in semantic diversity and outperforming temperature sampling in most settings. The method is validated by an ablation that reverses the merge order, supporting the hypothesis about layer‑specific diversity contributions.

By Yasuaki Sumita, Tomoharu Iwata, Toshiyuki Tanaka
arXiv Machine Learning
Oct 19

Can the Parts Fool the Test? Counterfactual Pairing Cycles for Relational OOD

The paper introduces counterfactual pairing cycles as a new evaluation method for detecting relational out-of-distribution errors in vision‑language models. By keeping the same two images and two captions but swapping their pairings, the approach neutralizes shortcuts based on image or text alone, enabling a cleaner assessment of true relational understanding. Experiments on synthetic data, GQA, and COCO show that this method distinguishes between general image‑caption matching and specific role‑binding failures, and that training with cycle losses improves relational accuracy.

By Yibo Zhou, Yirong Xiang
arXiv Machine Learning
Oct 19

The Loss or the Architecture? What Matters in Survival Analysis with Competing Risks

The paper investigates whether the choice of loss function or network architecture is more critical for survival analysis with competing risks. By independently varying three loss functions (two strictly proper) and four backbone architectures across 20 tabular and 3 pathology‑image datasets, the authors find that modelling cumulative incidence directly with censoring weighting consistently outperforms other losses, while the backbone must match the data modality. Proper scoring rules reveal that the loss dominates performance on competing‑risk data, whereas the architecture matters more on single‑event data, and the two factors compound rather than substitute.

By Julie Alberge, Ga\"el Varoquaux, Judith Ab\'ecassis
arXiv Machine Learning
Oct 19

Sentence-Level Topic Modeling with Lexical and Contextual Evidence

SentenceETM is a neural topic model that assigns a topic to each sentence by jointly using lexical and contextual evidence. It represents each topic with a vocabulary distribution via word embeddings and a Gaussian over sentence embeddings in a topic‑specific low‑dimensional subspace. The model exactly marginalizes the sentence topic, combining word and embedding evidence, and uses amortized variational inference for document proportions while re‑estimating Gaussian parameters in closed form.

By Federica Granese, Serena Villata, Charles Bouveyron
arXiv Machine Learning
Oct 19

Adaptive conformal prediction on the Grassmannian

The paper presents a conformal prediction framework for regression problems where the responses lie on the Grassmann manifold. It builds prediction regions as metric balls on the manifold, using basis‑invariant geometric distances as non‑conformity measures and scaling them with a learned prediction‑difficulty estimator to achieve locally adaptive, input‑dependent regions. The method preserves finite‑sample marginal coverage under exchangeability and is evaluated on synthetic data and an airfoil inverse shape‑design task, showing that geometry‑aware scores and local adaptation improve coverage uniformity and can reduce region size in heteroscedastic settings.

By Dimitrios Loukrezis, Zachary J. Grey, Dimitris G. Giovanis
arXiv Machine Learning
Oct 19

HardEval: An Execution-Verified Benchmark for AI-Driven Security Hardening and Compliance Remediation

HardEval is an execution‑verified benchmark that tests language‑model‑assisted security hardening on a live Red Hat Enterprise Linux 8 system. It uses STIG controls that fail on an unhardened baseline, and models generate Bash scripts that are run on a virtual machine and graded by OpenSCAP/OVAL compliance checks. The benchmark reveals significant differences in model performance across security domains and highlights failures where code appears plausible but does not meet the exact compliance condition.

By Manohar Vellala, Soumya Banerjee, Sandip Roy, Ross Gore, Sachin Shetty
arXiv Machine Learning
Oct 19

Beyond the Answer: Student-Conditioned Trace Construction for Reasoning Distillation

The paper introduces ReDiST, a framework that constructs student‑conditioned reasoning traces to enhance learning beyond answer‑only supervision. ReDiST revises student attempts using evidence and a verified answer, then selects the best traces with a frozen reader that evaluates answer recovery and probability. The distilled student is trained offline, and experiments on QA benchmarks show improved performance over baseline trace methods and highlight varying effectiveness across source and transfer settings.

By Shaoang Li, Yanhang Shi, Yufei Li, Mingfu Liang, Frank Shyu, Sandeep Pandey, Luke Simon, Xi Liu, Jian Li
arXiv Machine Learning
Oct 19

Accelerated Execution of Bayesian Neural Networks using a Single Probabilistic Forward Pass and Code Generation

The paper presents a pipeline for training, compiling, and deploying Bayesian neural networks (BNNs) that use a single probabilistic forward pass (PFP) instead of multiple stochastic samples. By assuming Gaussian-distributed weights and activations, PFP enables fully analytic uncertainty propagation, allowing a single deterministic forward pass to replace sampling. Using the TVM compiler and a library of Gaussian-propagating operators, the authors achieve up to 4200× speedups on small mini-batches while maintaining accuracy, uncertainty estimation, and out-of-domain detection comparable to traditional stochastic variational inference (SVI).

By Bernhard Klein, Falk Selker, Hendrik Borras, Sophie Steger, Franz Pernkopf, Holger Fr\"oning
arXiv Machine Learning
Oct 19

Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization

The paper introduces Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization, arguing that average treatment effect estimates of acquisition strategies overlook critical conditional effects tied to prior quality, budget ratio, and metric. An audit of 40 transfer-BO studies shows that 98% never vary the budget-to-algorithm ratio, leading to unstable rankings—e.g., Greedy outperforms UCB at a low budget but not at a higher one. The authors propose the Portable Regime Score (PRS) to capture this transition, demonstrate its predictive power across 79 diverse conditions, and present RegimePlanner, an online method that adapts acquisition based on PRS and achieves superior performance in HPO-B and GDSC2 benchmarks. whyItMatters":"The study highlights that unconditional rankings of Bayesian optimization methods are unreliable, emphasizing the need to report regime-specific metrics like B/|A| and prior rank correlation to accurately assess acquisition performance."

By Noel Thomas
arXiv Machine Learning
Oct 19

RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards

RubricEM is a unified reinforcement learning framework designed for deep research agents that plan, search, evaluate evidence, and synthesize long‑form reports. It introduces rubric‑guided policy decomposition, combining supervised fine‑tuning to scaffold agents with rubrics, stage‑structured GRPO for fine‑grained credit assignment, and a reflection meta‑policy for experience reuse. The resulting RubricEM‑8B model outperforms larger open models on four long‑form research benchmarks and approaches the performance of proprietary deep‑research systems.

By Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang, Jun Yan, Yanfei Chen, Chun-Liang Li, Long T. Le, Rujun Han, George Lee, Hanghang Tong, Chen-Yu Lee, Tomas Pfister
arXiv Machine Learning
Oct 19

OrbiSim: World Models as Differentiable Physics Engines for Embodied Intelligence

OrbiSim is a new robotic simulation paradigm that treats world models as a fully differentiable physics engine for embodied intelligence. It integrates structured scene assets, neural dynamics, and reinforcement learning into a single, physically grounded, end‑to‑end differentiable pipeline. The approach enables tasks such as differentiable contact modeling, gradient‑based policy optimization with sparse rewards, and intuitive physical inference, and it outperforms state‑of‑the‑art world models in predictive fidelity and control performance.

By Jiajian Li, Jingyuan Huang, Junru Gong, Qi Wang, Xiaokang Yang, Yunbo Wang
arXiv Computation and Language
Oct 19

Pearl-Chat: Engineering a native Luganda language model from first principles in JAX and Flax NNX

Pearl-Chat is an 8.5‑million‑parameter causal decoder‑only transformer built from first principles in JAX and Flax NNX, specifically engineered for the low‑resource Bantu language Luganda. It uses a byte‑level BPE tokenizer trained from scratch on curated Luganda corpora, achieving a two‑fold compression improvement over larger models like LLaMA‑3 and Gemma. The model is trained in two stages—pre‑training on diverse Luganda text and instruction fine‑tuning on dialogues—reaching a validation perplexity of 1.20 and high ROUGE‑L and BLEU scores on a held‑out diagnostic benchmark, with all weights, tokenizers, and code released openly.

By Wesley Kambale, Roland Sankara
arXiv Computation and Language
Oct 19

The Model Knows Better: Defending Medical Multi-Agent Systems Against Prompt Injection

The paper investigates prompt injection attacks on medical multi‑agent systems that use large language models (LLMs) for clinical decision support. Across two systems, two model families, and three benchmarks, a single injected passage can cause incorrect decisions in up to 61% of cases, and existing defenses only partially mitigate the threat. The authors propose a training‑free defense that verifies each clinical judgment against the patient’s findings, which reduces attack success on MedQA to 4‑7% while preserving accuracy on non‑attacked cases.

By Fardin Ahsan Sakib, Ziwei Zhu, Ozlem Uzuner
arXiv Computation and Language
Oct 19

Broad and Efficient PII Detection with Unified Labels

The paper introduces a 141‑million‑parameter encoder‑based PII detector trained on 1.12 million examples from 77 public datasets, all relabeled under a unified definition of broad contextual detection for 12 structured PII categories. It achieves higher recall than larger baselines on the AI4Privacy and Nemotron‑PII benchmarks at low false‑positive rates, while delivering 2–10× higher throughput and 94–97% less peak GPU memory usage. The model’s gains include improved coverage of multilingual disclosures and IP addresses embedded in URLs.

By Andy Hsu, Konstantin Berlin