Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning
Sep 14

Code-to-Harness: Distilling Black-Box Optimizers from Self-Play

The paper investigates whether an agent can learn a numerical search strategy through executable practice and then encode that strategy as text. By repeatedly writing and evaluating optimizer programs, the agent distills a 197‑word text called Harness A, which significantly reduces regret for Gemini Flash and other language‑model executors, matching the performance of classical Gaussian‑process Bayesian optimization. An independent replication produced a different but equally effective text, Harness B, and the framework also achieved the lowest regret on a sealed YouTube reward‑tuning benchmark.

By Yi Wu, Zheng Ren, Zhiyu Hu, Haochen Wang, Daryl Chang, Li Wei, Ting Wang, Zhen Li, Pooja Gupta, Nitin Jindal, Lukasz Heldt
arXiv Machine Learning
Sep 14

GLaMoR: Consistency Checking of OWL Ontologies using Graph Language Models

The paper introduces GLaMoR, a reasoning pipeline that converts OWL ontologies into graph-structured data and applies a Graph Language Model (GLM) for consistency checking. It addresses the challenge of verifying ontology consistency, especially for large ontologies where classical reasoners become computationally expensive. Experiments on NCBO BioPortal ontologies show that GLaMoR achieves 95% accuracy and is 20 times faster than traditional reasoners.

By Justin M\"ucke, Ansgar Scherp
arXiv Machine Learning
Sep 14

When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMs

The paper examines a specific type of hallucination in large language models caused by spurious correlations—unintended, statistically prominent associations in training data such as surnames linked to nationalities. These hallucinations are confidently produced, persist regardless of model scaling or refusal fine‑tuning, and evade existing detection methods like confidence filtering and inner‑state probing. The authors use controlled synthetic experiments and evaluations on both open‑source and proprietary LLMs, including GPT‑5, to demonstrate the failure of current detection techniques and provide a theoretical explanation for why statistical biases undermine confidence‑based approaches.

By Shaowen Wang, Yiqi Dong, Ruinian Chang, Tansheng Zhu, Yuebo Sun, Kaifeng Lyu, Jian Li
arXiv Machine Learning
Sep 14

Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting

The paper introduces Internal Dual-Wiener routing (Internal‑DW), a backward‑only method that weight‑balances internal gradient routes in autoregressive forecasting. By estimating bounded Wiener gains for identity and nonlinear paths, it suppresses unpredictable noise while preserving predictable learning signals, reducing forecast error by 5.2%–13.8% on four weak‑drive testbeds compared to full BPTT and outperforming gradient clipping, Jacobian regularization, and truncated BPTT in most cases. The approach shows that long‑horizon supervision can be effective without trusting every backward gradient equally.

By Junhao Zhao, David Michael Simberg, Jacob Kang, Colin Connor Kurniawan, Nan Xu
arXiv Machine Learning
Sep 14

DynSHAP: Towards Explainable Dynamic Survival Analysis

DynSHAP is a SHAP-based framework designed for dynamic survival analysis, extending marginal SHAP estimators to handle time–feature pairs as players in the Shapley game. It introduces Temporal DynSHAP, which models linear dependencies across time and uses conditional sampling to improve explanations. Experiments on synthetic data and two real-world clinical datasets show that Temporal DynSHAP more accurately recovers temporally dependent features and provides faithful attributions for two DSA architectures, enabling medical experts to identify which patient information influenced predictions and when.

By Nastasya Anokhina, Jonas J\"ur{\ss}, Pietro Li\`o
arXiv Computation and Language
Sep 14

LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?

LLM‑BabyBench transforms the BabyAI gridworld into a fully observable, purely textual setting that isolates planning as the sole source of failure. By serialising the entire grid, providing formal instructions, and validating actions deterministically, the benchmark introduces the PPD suite—Predict, Plan, and Decompose tasks—each scored with metrics that separate mission understanding from sequencing. Across a range of large language models, simulation accuracy is high while planning success drops sharply beyond a model‑specific horizon, revealing that plan length—not grid size—drives failure and that models often commit to a single corridor‑shaped route without backtracking.

By Idriss Malek, Omar Choukrani, Daniil Orel, Anh Duy Le Dinh, Zhuohan Xie, Zangir Iklassov, Martin Tak\'a\v{c}, Salem Lahlou
arXiv Computer Vision
Sep 14

No One Knows the State of the Art in Geospatial Foundation Models

The paper "No One Knows the State of the Art in Geospatial Foundation Models" critiques the current lack of standardization in geospatial foundation model (GFM) research, highlighting inconsistencies in evaluation, training, and model release practices across 152 papers. It reports significant discrepancies—46 cross-paper disagreements of at least 10 points for the same model and benchmark, 94 out of 126 papers using unique pretraining configurations, and 39% of papers releasing no model weights. The authors propose six concrete expectations, including named-license weight release, shared core evaluations, and a unified evaluation harness, to address these coordination failures and foster a clearer, comparable understanding of GFM progress.

By Isaac Corley, Nils Lehmann, Caleb Robinson, Gabriel Tseng, Anthony Fuller, Hamed Alemohammad, Evan Shelhamer, Jennifer Marcus, Hannah Kerner
arXiv AI
Sep 12

Relevance Is Not Permission: Localizing and Controlling Metric-Facing Attention Contributions

The paper introduces Warrant, a method that locates and controls the contributions of attention mechanisms to model metrics. Warrant exposes the item‑wise contribution path to the reported metric and applies query‑conditioned permission on that path. Experiments on multiple datasets show that Warrant improves primary metrics in most comparisons, reveals a weak correlation between attention and prediction utility, and demonstrates that learned permission can recover evidence ranking while suppressing distractors.

By Minwoo Yu, Young-guk Ha
arXiv AI
Sep 12

Probabilistic Focal Search: Accelerating Bounded-Suboptimal Search via Lower-Bound Advancement

Probabilistic Focal Search (PFS) augments traditional Focal Search by probabilistically choosing between the standard heuristic-guided expansion and expanding the minimum‑f node in OPEN. This strategy advances the lower bound, enlarges the FOCAL frontier, and can dramatically reduce node expansions—up to 90% in some benchmarks such as N‑Puzzle and TSP—especially when long f_min plateaus delay useful FOCAL admissions. An anytime variant, APFS, outperforms other tested anytime algorithms on the Generalized Covering TSP, and the same probabilistic scheduler transfers to Dynamic Potential Search as Probabilistic Dynamic Potential Search (PDPS), though its effectiveness varies by domain and bound.

By Minh Vu Duc, Trung Le Huu, H\`a Minh Ho\`ang, Trung Thanh Nguyen, Phuong Khanh Nguyen, Huynh Thi Thanh Binh
arXiv AI
Sep 12

Automating Quadratic Unconstrained Binary Optimization (QUBO) Formulation Generation from Natural Language

The paper introduces an end‑to‑end multi‑agent framework that automatically converts natural‑language problem descriptions into Quadratic Unconstrained Binary Optimization (QUBO) formulations, a key format for combinatorial optimization on quantum and hybrid solvers. It also presents QUBOBench, a benchmark of 100 problems from 12 domains, and reports that the framework achieves 68% accuracy—22% better than a single‑call baseline—thanks largely to iterative self‑repair. The authors provide open‑source data and code at https://quitttcat.github.io/QuantumQUBOAgent.

By Niloy Kumar Mondal, Md Rizwan Parvez
arXiv AI
Sep 12

Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning

The paper introduces belief‑shift branching, a method for placing forks in tree‑structured reinforcement learning rollouts by detecting where a model’s answer belief changes most sharply. Unlike traditional fixed‑length or entropy‑based forking, this approach uses a lightweight probe or learned activation direction to identify pivots in the value curve, reducing unnecessary sampling. Experiments show that belief‑shift forking consistently outperforms baseline methods across multiple models and benchmarks, yielding significant gains in mathematics and code tasks.

By Bin Lei, Yu Li, Prafulla Kumar Choubey, Jiaxin Zhang, Becky Xiangyu Peng, Qinyuan Ye, Kartik Narayan, Caiwen Ding, Silvio Savarese, Chien-Sheng Wu