Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv Machine Learning
Sep 14

When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMs

The paper examines a specific type of hallucination in large language models caused by spurious correlations—unintended, statistically prominent associations in training data such as surnames linked to nationalities. These hallucinations are confidently produced, persist regardless of model scaling or refusal fine‑tuning, and evade existing detection methods like confidence filtering and inner‑state probing. The authors use controlled synthetic experiments and evaluations on both open‑source and proprietary LLMs, including GPT‑5, to demonstrate the failure of current detection techniques and provide a theoretical explanation for why statistical biases undermine confidence‑based approaches.

By Shaowen Wang, Yiqi Dong, Ruinian Chang, Tansheng Zhu, Yuebo Sun, Kaifeng Lyu, Jian Li
arXiv Machine Learning
Sep 14

Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting

The paper introduces Internal Dual-Wiener routing (Internal‑DW), a backward‑only method that weight‑balances internal gradient routes in autoregressive forecasting. By estimating bounded Wiener gains for identity and nonlinear paths, it suppresses unpredictable noise while preserving predictable learning signals, reducing forecast error by 5.2%–13.8% on four weak‑drive testbeds compared to full BPTT and outperforming gradient clipping, Jacobian regularization, and truncated BPTT in most cases. The approach shows that long‑horizon supervision can be effective without trusting every backward gradient equally.

By Junhao Zhao, David Michael Simberg, Jacob Kang, Colin Connor Kurniawan, Nan Xu
arXiv Machine Learning
Sep 14

DynSHAP: Towards Explainable Dynamic Survival Analysis

DynSHAP is a SHAP-based framework designed for dynamic survival analysis, extending marginal SHAP estimators to handle time–feature pairs as players in the Shapley game. It introduces Temporal DynSHAP, which models linear dependencies across time and uses conditional sampling to improve explanations. Experiments on synthetic data and two real-world clinical datasets show that Temporal DynSHAP more accurately recovers temporally dependent features and provides faithful attributions for two DSA architectures, enabling medical experts to identify which patient information influenced predictions and when.

By Nastasya Anokhina, Jonas J\"ur{\ss}, Pietro Li\`o
arXiv Computation and Language
Sep 14

LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?

LLM‑BabyBench transforms the BabyAI gridworld into a fully observable, purely textual setting that isolates planning as the sole source of failure. By serialising the entire grid, providing formal instructions, and validating actions deterministically, the benchmark introduces the PPD suite—Predict, Plan, and Decompose tasks—each scored with metrics that separate mission understanding from sequencing. Across a range of large language models, simulation accuracy is high while planning success drops sharply beyond a model‑specific horizon, revealing that plan length—not grid size—drives failure and that models often commit to a single corridor‑shaped route without backtracking.

By Idriss Malek, Omar Choukrani, Daniil Orel, Anh Duy Le Dinh, Zhuohan Xie, Zangir Iklassov, Martin Tak\'a\v{c}, Salem Lahlou
arXiv Computer Vision
Sep 14

No One Knows the State of the Art in Geospatial Foundation Models

The paper "No One Knows the State of the Art in Geospatial Foundation Models" critiques the current lack of standardization in geospatial foundation model (GFM) research, highlighting inconsistencies in evaluation, training, and model release practices across 152 papers. It reports significant discrepancies—46 cross-paper disagreements of at least 10 points for the same model and benchmark, 94 out of 126 papers using unique pretraining configurations, and 39% of papers releasing no model weights. The authors propose six concrete expectations, including named-license weight release, shared core evaluations, and a unified evaluation harness, to address these coordination failures and foster a clearer, comparable understanding of GFM progress.

By Isaac Corley, Nils Lehmann, Caleb Robinson, Gabriel Tseng, Anthony Fuller, Hamed Alemohammad, Evan Shelhamer, Jennifer Marcus, Hannah Kerner
arXiv AI
Sep 12

Relevance Is Not Permission: Localizing and Controlling Metric-Facing Attention Contributions

The paper introduces Warrant, a method that locates and controls the contributions of attention mechanisms to model metrics. Warrant exposes the item‑wise contribution path to the reported metric and applies query‑conditioned permission on that path. Experiments on multiple datasets show that Warrant improves primary metrics in most comparisons, reveals a weak correlation between attention and prediction utility, and demonstrates that learned permission can recover evidence ranking while suppressing distractors.

By Minwoo Yu, Young-guk Ha
arXiv AI
Sep 12

Probabilistic Focal Search: Accelerating Bounded-Suboptimal Search via Lower-Bound Advancement

Probabilistic Focal Search (PFS) augments traditional Focal Search by probabilistically choosing between the standard heuristic-guided expansion and expanding the minimum‑f node in OPEN. This strategy advances the lower bound, enlarges the FOCAL frontier, and can dramatically reduce node expansions—up to 90% in some benchmarks such as N‑Puzzle and TSP—especially when long f_min plateaus delay useful FOCAL admissions. An anytime variant, APFS, outperforms other tested anytime algorithms on the Generalized Covering TSP, and the same probabilistic scheduler transfers to Dynamic Potential Search as Probabilistic Dynamic Potential Search (PDPS), though its effectiveness varies by domain and bound.

By Minh Vu Duc, Trung Le Huu, H\`a Minh Ho\`ang, Trung Thanh Nguyen, Phuong Khanh Nguyen, Huynh Thi Thanh Binh
arXiv AI
Sep 12

Automating Quadratic Unconstrained Binary Optimization (QUBO) Formulation Generation from Natural Language

The paper introduces an end‑to‑end multi‑agent framework that automatically converts natural‑language problem descriptions into Quadratic Unconstrained Binary Optimization (QUBO) formulations, a key format for combinatorial optimization on quantum and hybrid solvers. It also presents QUBOBench, a benchmark of 100 problems from 12 domains, and reports that the framework achieves 68% accuracy—22% better than a single‑call baseline—thanks largely to iterative self‑repair. The authors provide open‑source data and code at https://quitttcat.github.io/QuantumQUBOAgent.

By Niloy Kumar Mondal, Md Rizwan Parvez
arXiv AI
Sep 12

Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning

The paper introduces belief‑shift branching, a method for placing forks in tree‑structured reinforcement learning rollouts by detecting where a model’s answer belief changes most sharply. Unlike traditional fixed‑length or entropy‑based forking, this approach uses a lightweight probe or learned activation direction to identify pivots in the value curve, reducing unnecessary sampling. Experiments show that belief‑shift forking consistently outperforms baseline methods across multiple models and benchmarks, yielding significant gains in mathematics and code tasks.

By Bin Lei, Yu Li, Prafulla Kumar Choubey, Jiaxin Zhang, Becky Xiangyu Peng, Qinyuan Ye, Kartik Narayan, Caiwen Ding, Silvio Savarese, Chien-Sheng Wu
arXiv AI
Sep 12

DRG-MAPPO: Hierarchical Dynamic Role-Graph Multi-Agent Reinforcement Learning for Cooperative Air Combat

The paper introduces DRG-MAPPO, a hierarchical multi‑agent reinforcement learning framework for cooperative air combat. It combines graph‑based relational modeling with dynamic role assignment, using a high‑level policy to allocate tactical roles such as leader and supporter, and a low‑level policy to execute maneuver actions. The approach includes a target‑priority auxiliary task and achieves an 87% win rate in experiments, indicating effective coordination and stability.

By Junlin Liu, Chengwei Li, Yang Gao, Hui Chang, Xinchen Zhang, Zhijun Zhao, Hao Zhao
arXiv AI
Sep 12

Debate-to-Skill: Capability-Bound Process Supervision for Industrial Query-to-Agent Annotation

The paper introduces Debate‑to‑Skill, a capability‑bound process supervision method for annotating industrial query‑to‑agent tasks. It addresses failures caused by confusing topical relevance with executable capability, especially for long‑tail and boundary‑sensitive requests. By employing reusable decision principles, structured deliberation, verifier‑based verdict extraction, and disagreement‑driven refinement, the approach is evaluated against direct‑label supervision, reasoning‑SFT, and structural ablations on a Query2Agent benchmark, focusing on grey‑zone cases where semantic relatedness and executable capability diverge.

By Shiyu Zhang, Leisheng Cheng, Huifu Li