Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Machine Learning
Oct 23

Continuous-depth Deep Gaussian Processes

The paper introduces a continuous-depth formulation of Deep Gaussian Processes (DGPs), treating depth as a latent continuous evolution rather than a discrete stack. It proposes two instantiations—CDGP, which applies a GP prior directly to the state–depth drift, and FlowDGP, which uses a flow-evolved prior to capture richer depth dependence at lower training cost. Experiments on synthetic, benchmark, and large-scale regression tasks demonstrate that residual DGP outperforms plain DGP, while CDGP and FlowDGP provide stronger overall performance across the studied regimes.

By Ying Li, Sanyou Wu, Zi Yang, Qiaochu Xu, Yuhao Liu, Zhidi Lin, Michael Minyi Zhang, Petar Djuric, Alexandre H. Thiery
arXiv Computer Vision
Oct 23

Ask About the Right Moment: Training-Free Object Identity Reasoning for Video-Language Models

The paper introduces a training‑free method called “Track, Then Bind” for video‑language models to reason about object identity over time. It separates semantic identity binding from temporal identity maintenance by using a promptable video tracker to follow candidate objects and a frozen vision‑language model to bind an object to a track based on visible evidence. The approach achieves state‑of‑the‑art performance on several benchmarks, outperforming fine‑tuned trajectory models and other training‑free agents on tasks such as VET‑Bench, Perception Test cup games, and TOC‑Bench.

By Di Yang, Xiaochang Li, Muyang Liu, Yufei Wang, Yanhai Xiong
arXiv Computation and Language
Oct 23

TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents

TauIndianBankBench is a new benchmark that evaluates conversational agents on 1,000 Indian retail‑banking tasks by checking not only the final records but also whether every required procedural step was performed and no unrequested changes were made. It highlights that models can appear successful when judged only on final records, yet may skip necessary steps or alter records incorrectly. The benchmark provides a more accurate assessment of language model performance in the Indian banking domain.

By Krishanu Adhikary, Prashant Devadiga
arXiv AI
Oct 23

SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation

SWE-DIVE is a dynamic, interactive framework for evaluating coding agents in multi-turn scenarios. It introduces a user simulator that generates varied behavioral patterns and a conversation validator that checks whether the simulator’s outputs meet initial requirements, guiding the agent toward task completion. The framework runs within a Harbor environment that orchestrates the agent, simulator, and validator to assess success on coding tasks.

By Lorenz Brehme, Benjamin Steenhoek, Ruth Breu
arXiv Machine Learning
Oct 23

IRAH: Hierarchical Agentic Tree Search for Retrieval Pipeline Optimization

The paper introduces IRAH, a method for optimizing Retrieval-Augmented Generation pipelines by combining a Language Agent Tree Search for IR (LATS-IR) with offline meta‑learned initialization. LATS‑IR performs a UCT‑guided hierarchical search, first exploring macro‑structural pipeline choices and then refining numerical parameters, while using natural‑language reflections to guide mutations and provide inspectable search traces. Evaluations on three in‑distribution and one out‑of‑distribution benchmarks show that IRAH, especially when warm‑started, achieves the highest average held‑out nDCG@10 (0.4816) and outperforms TPE on two of the three in‑distribution benchmarks.

By Maria Ana Cardei, Aniket Singh, Amir Moghadam, Srividhya Baikadi, Rashmi Shetty
arXiv AI
Oct 23

GenXR-Bench: Benchmarking Coding Agents for Interactive XR Development

GenXR-Bench is a new benchmark designed to evaluate coding agents in the context of interactive extended reality (XR) development. It comprises 112 expert-authored tasks—including feature implementations, bug repairs, and application generation—alongside a testing suite that uses scripted and adaptive interactions with programmatic and visual assertions. The benchmark achieved 96.5% accuracy against expert judgments on 124 candidate applications, yet current models only reach a 50.9% Pass@1 score, highlighting challenges such as asynchronous sensor lifecycles and embodied interaction semantics.

By William Huang, David Li, Xiang 'Anthony' Chen, Xun Qian, Nels Numan, Zhongyi Zhou, Hyunsung Cho, Salman Muin Kayser Chishti, Ding Xia, Xincheng Huang, Jiahao Ren, Faraz Faruqi, Tianyu Xu, Yang Zhang, David Kim, Ruofei Du
arXiv AI
Oct 23

RegoMender: A Multi-agent Framework for Automated Rego Policy Repair

RegoMender is a multi‑agent framework that automates the repair of Rego policies, a key component of Policy‑as‑Code in cloud‑native systems. It uses a shared blackboard and atomic toolkit to dynamically apply patching, rewriting, or rollback actions, achieving high syntactic (82.9%) and semantic (77.9%) success rates on the newly introduced RegoBench benchmark of 1,159 real‑world defects. The system also generalizes to other languages such as HCL and SQL, repairing 86.4% of 132 real‑world CVE vulnerabilities, thereby demonstrating robustness and practical security value.

By Ruijie Zhang, Huaxiao Yin, Li Hu, Fanqi Kong, Haozhe Liang, Shaoyin Cheng, Kun Zhang, Bibo Tu
arXiv AI
Oct 23

PaceVLA: Adaptive Temporal Action Quantization in Vision-Language-Action Models

PaceVLA introduces adaptive temporal action quantization for Vision‑Language‑Action models, enabling faster robot execution by aggregating low‑level actions into coarser steps when appropriate. The framework combines mode‑specific action decoders with a quantizability‑guided mode selection and a context‑conditioned router to decide execution granularity at each inference step. Experiments on simulation benchmarks and real‑world tasks demonstrate that PaceVLA improves execution efficiency while preserving task success rates across diverse embodiments.

By Hojin Jeon, Jimin Lee, Taekwan Kim, Nayoung Oh, Taeyoung Kim, Myungkyu Koo, Jinmo Koo, Joonwoo Ahn, Jinwoo Shin
arXiv Machine Learning
Oct 23

RiboBridge: protein-conditioned de novo RNA binder structure design

RiboBridge is a new structure-first framework for designing RNA molecules that bind to target proteins without assuming a predefined binding site. It separates RNA conformational generation from protein–RNA assembly by first proposing candidate binding interfaces on the protein surface, generating canonical RNA conformations in a pretrained RNA latent space (RiboSphere), and then predicting interaction constraints to bridge the RNA and protein coordinate frames. The predicted interaction geometry is converted into a geometrically consistent protein-relative pose via rigid placement, and RiboBridge outperforms existing methods on a benchmark of 124 protein–RNA complexes, achieving higher fNAT and DockQ scores and lower RMSDs.

By Zhou Zhang, Hanqun Cao, Minghao Sun, Odin Zhang, Yang Zhang, Pheng Ann Heng, Wujun Li, Tianfan Fu
arXiv Machine Learning
Oct 23

Lean Finder V2: Semantic Search for Mathlib That Disambiguates User Queries

Lean Finder V2 is a semantic search framework for Mathlib that resolves ambiguous user queries by combining adversarial training at the model level with input-level query rewriting. It addresses the common scenario where mathematicians and coding agents issue short, context‑poor queries, which existing search engines struggle to interpret. On the MathlibQR benchmark, Lean Finder V2 achieves at least an 11% relative improvement over prior engines and further boosts retrieval performance through optimized query rewriting, while also enhancing downstream theorem‑proving success when integrated with Claude Code.

By Jialin Lu, Xiaoyu Liu, Hengrui Luo, Rodrigo Stehling, Linxi Liu, Weiran Sun, Yuepeng Wang, Wuyang Chen
arXiv Machine Learning
Oct 23

SQuAT: Spherical Lattice Vector Quantization-Aware Training

SQuAT introduces a shape‑gain vector quantization‑aware training (VQ‑QAT) method that separates weight direction and magnitude, using an $E_8$‑derived spherical codebook for directions and low‑bit scalar quantizers for gains. By applying distinct straight‑through estimators (STE) to each component, SQuAT induces tangent‑space updates that act as an implicit spherical (Riemannian) optimization, improving training dynamics over standard STE approaches. A distributed exact nearest‑neighbor assignment scheme further reduces training time by 12.8×, enabling scaling to 10 B tokens and achieving state‑of‑the‑art accuracy on several large language models.

By Sebastian Loeschcke, Ivan Koryakovskiy, Tycho F. A. van der Ouderaa, Riccardo Del Chiaro, Mart van Baalen, Paul N. Whatmough, Markus Nagel
arXiv Machine Learning
Oct 23

TerraTransfer++: Demonstration-Free End-to-End Planner

TerraTransfer++ demonstrates a demonstration‑free, end‑to‑end autonomous driving framework that leverages self‑play in vectorized simulators to pretrain a policy. The method distills self‑play rollouts into a flow‑matching trajectory teacher and trains lightweight adapters on a frozen vision backbone, using action KL divergence, a batch‑relational low‑rank loss, and trajectory‑matching loss. The resulting closed‑loop policy surpasses all published imitation‑based methods on the HUGSim benchmark, while the planner achieves state‑of‑the‑art performance on NAVSIM navhard and navtest without any human demonstrations.

By Zikang Xiong, Weixin Li, Zhouchonghao Wu, Akshay Rangesh, Saarth Bonde, Grantland Hall, Chen Tang, Yihan Hu, Wei Zhan
arXiv Machine Learning
Oct 23

Flow-based Spectral Kernel Learning for Nonstationary Attention

The paper introduces Flow-based Spectral Kernel Attention (FSKA), a framework that learns nonstationary attention kernels using spectral density learning and normalizing flows. FSKA replaces traditional query–key projections with a simplified design that retains fixed orthogonal value projections, leading to a more parameter‑efficient architecture. Experiments on classification tasks demonstrate competitive or improved performance while reducing trainable projection parameters by roughly 40%–47% and preserving linear‑attention memory scaling.

By Zi Yang, Ying Li, Alejandro Lancho, Dar\'{i}o C. Larese, Pablo M. Olmos, Michael Minyi Zhang
arXiv Machine Learning
Oct 23

Diagnostic Loss: Asymmetric Supervision for Radiology Report Generation

The paper introduces Diagnostic Loss (DLoss), an objective designed to align radiology report generation with the asymmetric requirement of accurately stating supported diagnoses while avoiding unsupported ones. DLoss operates on decoder token probabilities, aggregating mass over vocabularies for supported and unsupported diagnoses and applying asymmetric pooling to encourage correct diagnoses and suppress spurious ones. Applied to various 3D MedVLMs on 50,000 abdominal CT scans, DLoss improves diagnostic faithfulness without compromising conventional language‑generation metrics.

By Ignacy Kolton, Wenxuan Li, Zongwei Zhou, Jaros{\l}aw B. \'Cwik{\l}a, Arkadiusz Sitek, Pedro R. A. S. Bassi, Szymon P{\l}otka
arXiv Machine Learning
Oct 23

Super Agents and Confounders: Influence of surrounding agents on vehicle trajectory prediction

The paper analyzes state‑of‑the‑art vehicle trajectory predictors and finds that many surrounding agents actually worsen prediction accuracy. Using Shapley attribution, it shows models learn unstable, non‑causal decision schemes that vary across training runs. The authors introduce a Conditional Information Bottleneck that compresses agent features and ignores unhelpful ones, improving accuracy and robustness across datasets and architectures.

By Daniel Jost, Luca Paparusso, Martin Stoll, J\"org Wagner, Raghu Rajan, Joschka B\"odecker
arXiv Machine Learning
Oct 23

When Should LLMs Reason? Adaptive Inference Routing via Entropy Dynamics

The paper introduces EDRM, an entropy‑dynamics‑based routing framework that decides whether to use direct, standard, or chain‑of‑thought decoding for large language models. By analyzing early decoding entropy, EDRM identifies tasks that benefit from explicit reasoning and routes queries accordingly, achieving 16–55% token savings while maintaining accuracy across 15 benchmarks and 7 LLMs. The study also shows that a short direct fallback branch can further improve accuracy by up to 4.9 percentage points compared to routing alone.

By Wei Xia, Haoqing Wang, Zhi-Hong Deng, Yehui Tang
arXiv Machine Learning
Oct 23

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-Based RL

The paper introduces Rubric Dropout, a technique that randomly removes a subset of rubric criteria during training to prevent reinforcement learning agents from exploiting the rubric as a proxy for quality. By resampling which criteria are rewarded at each step and ensuring all responses to a prompt are evaluated on the same subset, the method reduces reward hacking and improves out‑of‑distribution performance on medical and science benchmarks. Block Dropout further groups semantically similar criteria to avoid near‑duplicate rewards, yielding additional gains.

By Minglai Yang, Xinyu Guo, Zhengliang Shi, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Dan Roth, Ying Liu
arXiv Machine Learning
Oct 23

Meta-learning of Gibbs states for many-body Hamiltonians with applications to Quantum Boltzmann Machines

The paper introduces two meta‑learning algorithms—Meta‑Variational Quantum Thermalizer (Meta‑VQT) and Neural Network Meta‑VQT (NN‑Meta VQT)—to efficiently prepare quantum Gibbs states for parametrized many‑body Hamiltonians on NISQ devices. Meta‑VQT employs a fully quantum ansatz, while NN‑Meta VQT uses a quantum‑classical hybrid architecture, both optimizing over a finite training set to generate accurate thermal states at unseen parameter values. The methods are validated on Transverse Field Ising Models up to eight qubits, a two‑qubit Heisenberg model, a 3‑qubit Kitaev ring, and are applied to train a Quantum Boltzmann Machine, yielding significant speedups and improved accuracy over existing techniques.

By Ruchira V Bhat, Rahul Bhowmick, Avinash Singh, Krishna Kumar Sabapathy
arXiv Computation and Language
Oct 23

What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks

The paper identifies a "specification-oracle gap" in coding‑agent benchmarks, where tests rely on details (constants, module paths, exact messages) that are not stated in the natural‑language task description. The authors formalise this gap, develop a static audit tool that classifies such dependencies across five languages, and demonstrate that many benchmarks (e.g., SWE‑Milestone) contain tasks whose tests depend on unstated information, leading to higher failure rates for agents that do not reproduce upstream implementation details. The study also shows that the prevalence of these gaps varies with benchmark construction and that removing affected tests does not alter leaderboard rankings but shifts scores within repositories, favoring agents that replicate upstream names. whyItMatters":"The findings reveal that benchmark tests can inadvertently measure an agent’s ability to mimic upstream implementation details rather than its true problem‑solving skills, potentially skewing evaluations and comparisons across agents."

By Fei Ding
arXiv Computation and Language
Oct 23

Spectral Smoothing and Spiking Perturbations Reconfigure Capability Profiles and Selective Task Performance in Large Language Models

The paper investigates how post‑hoc spectral smoothing and spiking interventions on transformer feed‑forward projection matrices affect large language models. By altering singular‑value spectra while preserving singular vectors and the geometric mean, the authors evaluate the impact on a range of tasks—including reasoning, truthfulness, hallucination, safety, and personality probes—across several Llama and Qwen models. The results reveal that structured perturbations can selectively suppress or enhance specific capabilities, suggesting a lightweight method for probing how different abilities are organized and traded off within these models.

By Qince Luan, Jianhui Xu, Yang Xu