Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 28

Don't Pick a Winner: Knowledge Conflicts Need a Prior, Not a Rule

The paper introduces a retrieval‑augmented language model that resolves disagreements between retrieved context and the model’s own knowledge by incorporating a prior over the reliability of the context rather than a fixed rule. Experiments on the GRACE benchmark show that this prior‑based approach outperforms existing baselines by up to 26 points, achieving higher overall scores on both Qwen3.5‑35B‑A3B and Llama‑3.1‑8B. The method also demonstrates superior performance on the ClashEval benchmark, confirming the theoretical advantage of evidence‑based conflict resolution.

By Minh Pham Van, Cong Tran
arXiv AI
Oct 28

kw-bench: Breaking Frontier Harnesses on Real-World Tasks

The paper introduces kw-bench, a benchmark that evaluates coding-agent harnesses on investigative knowledge work, specifically a five‑year VAT reconstruction task across multiple SAP systems. It contrasts generative benchmarks with this real‑world, evidence‑heavy task, showing that model‑agnostic harnesses performed well while lab harnesses failed. The study demonstrates how frontier models can dramatically reduce time and cost for complex audit‑style engagements.

By George Juraj Salapa
arXiv AI
Oct 28

RouteDRAG: Learning to Route and Control Agentic Execution for Multimodal Document RAG

RouteDRAG is a learning-based framework for multimodal document retrieval-augmented generation that learns when to invoke an agent and how to control its execution. It comprises a unified document index, adaptive path routing, dynamic agent execution, and sequential policy learning to balance accuracy and computational cost. Experiments on two benchmarks show a 3.47‑point accuracy gain over shorter‑path-only methods while using the agent for only 79.3% of queries, outperforming uniform agent execution with fewer answering calls.

By Zekai Chen, Hao Zhu, Yu Liu, Xunkai Li, Zejian Kang, Rui Qian, Quanjiang Li, Rong-Hua Li
arXiv AI
Oct 28

AC-JEPA: Anchored Cross-Modal Joint Embedding Predictive Architecture for Visual-to-Symbolic Translation

AC-JEPA is a plug‑in for supervised fine‑tuning that predicts masked intermediate states in both directions of a cross‑modal translation task, combining a joint‑embedding objective with causal token‑level supervision. It is designed to be robust to imperfect paired data by learning from continuous latent targets derived from pseudo‑labels, rather than hard categorical supervision. In optical chemical structure recognition, AC‑JEPA outperforms plain supervised fine‑tuning, achieving 76.07 % accuracy on clean pairs and 80.23 % when self‑training with pseudo‑labels, whereas SFT alone drops to 63.47 % under the same conditions.

By Chaofei Hong, Ziyang Chen, Baochen Wang, Christina Yi Jin, Qing Zhang
arXiv AI
Oct 28

ATEMPO: Amortized Transfer Entropy Model Per-lag Over-Time

ATEMPO is a diffusion-based transfer entropy estimator that amortizes over lags and time, allowing a single score network to estimate the full TE surface without separate fits for each source and conditioning depth. By sampling depths during training and conditioning on temporal position, ATEMPO recovers when information arrives and achieves high lag-detection accuracy on synthetic and real datasets. The method improves downstream forecasting and brain-computer interface performance, reducing MSE on WEATHER-5K and increasing mean R² by 4.1% compared to competing baselines.

By Emma Andrews, Konstantinos Kontras, Keane Ong, Maarten De Vos, Gianmarco Mengaldo
arXiv AI
Oct 28

ADTGuard: Adaptive Depth-Time Safeguard for Streaming Inference in Large Language Models

ADTGuard is a new safeguard for large language models that monitors each token before it is streamed to users. It uses Depth-Time Safety Dynamics to combine safety signals from multiple layers and a Causal Depth Router to select the most relevant layers for each prefix. With only response‑level labels, it refines token‑level training targets and achieves strong streaming detection performance with minimal parameters and latency.

By Chuming Wei, Churui Yang, Zihan Zhang, Ruien Li, Wusheng Zhang, Yan Zhang, Guangwen Yang
arXiv AI
Oct 28

Beyond the Evidence Coherence Trap: Agent-Driven Evidence Selection for Temporally Disordered Fake News Detection

The paper introduces ChronoFake, a benchmark of 30,706 news articles that isolates temporally disordered fake news, where factual events are rearranged into false chronologies. It identifies the "evidence coherence trap," where verification relies on locally coherent but temporally mismatched evidence. To counter this, the authors propose DelSA, a document-selection agent that iteratively refines evidence during training and uses a frozen rulebook for inference, leveraging an LLM for evidence characterization and a verifier-trained reranker for timeline consistency, achieving superior performance on ChronoFake and competitive results on other datasets.

By Chi Wang, Min Gao, Yifei Zhang, Bincheng Gu, Zongwei Wang, Wentao Li
arXiv AI
Oct 28

Evolving Agent Teams

Evolving Agent Teams (EAT) is a multi‑agent framework designed to overcome local optima that plague existing coding agents when optimizing GPU kernels. It employs a central supervisor to diagnose and restructure agent teams and a persistent file system that records learning documents, evolution history, and team artifacts. Across a range of benchmarks—including FlashInfer, FlashKDA, and RoboDojo—EAT consistently outperforms strong coding‑agent baselines, achieving up to 1.2× speedup on FlashInfer kernels and superior results on several other tasks.

By Shiyi Cao, Ziming Mao, Dacheng Li, Joseph E. Gonzalez, Ion Stoica
arXiv AI
Oct 28

Proactive Memory for Event-Driven LLM Agents

Proactive Memory for Event-Driven LLM Agents introduces ProactMem, a system that organizes an agent’s history into recent events, temporal trajectories, and behavioral regularities. It validates trajectory facts against source events and assesses regularities through historical replay before matching them to current conditions. The authors also present ProactiveStream, a longitudinal benchmark with three application scopes, and report that ProactMem achieves an overall F1 of 52.6%, a 21.4% relative improvement over the strongest of five external baselines.

By Jinyang Li, Zichuan Fu, Xian Wu, Wenlin Zhang, Yimin Deng, Kaifeng Guo, Guojing Li, Junjia QI, Xiangyu Zhao
arXiv AI
Oct 28

AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations

AutoVend is a benchmark that tests large language model agents by having them manage 20 vending machines across five site types over 180 simulated days via a web interface. The simulator incorporates realistic logistics such as shared warehouse inventory, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. Five LLM agents were evaluated against a rule‑based baseline, revealing that most agents underperformed the baseline and that higher inference costs can distort performance rankings.

By Qiming Shi, Yikun Li, Beidi Luan, Rui Sun, Yan Ma, Jing Li, Daxin Jiang, Zuo Bai
arXiv AI
Oct 28

Guided-RLVR: Guided Exploration in Reinforcement Learning with Verifiable Rewards

Guided-RLVR introduces a lightweight controller to reinforcement learning with verifiable rewards (RLVR), injecting semantically meaningful steering tokens into the sampling distribution. This strategy encourages large language models to explore multiple reasoning perspectives, expanding the set of reasoning trajectories. Experiments on mathematical reasoning and code generation benchmarks demonstrate that Guided-RLVR consistently outperforms existing RLVR methods while producing more diverse reasoning paths.

By Bo Yang, Zishuai Zhang, Weihua Li, Qinnan Zhang, Sijia Wen, Hainan Zhang
arXiv AI
Oct 28

Correctness Conditioning as Selection in Chain-of-Thought Faithfulness Evaluation

The paper investigates how restricting chain‑of‑thought faithfulness evaluation to only correctly answered instances alters the measured population. Using a seven‑variable structural causal model, the authors derive an exact expression for the observed gap and distinguish between difficulty confounding and shortcut‑induced collider bias. A review of 18 papers and empirical evaluation on arithmetic tasks reveal positive selection gaps, with the GLM‑5.3 run showing a +7.4 percentage‑point increase and a pooled analysis yielding an odds ratio of 1.70.

By Yueer Pan, Yuexi Pan
arXiv AI
Oct 28

Does Your Generalist Skill Optimizer Really Incentivize Agentic Tasks? Rethinking Skill Optimization via Bipartite Attribution

The paper introduces BiPA, a method for evaluating skill optimizers that rewrite natural‑language procedures for agents. By replaying the optimizer’s acceptance rule on identical candidates and recording episodes as bipartite graphs of turns and read text, BiPA isolates the effect of selection from the cost of added input tokens. Experiments on three optimizers, three benchmarks, and two models show that learned skills perform only marginally better than hand‑written seeds, and that apparent dev‑set gains often disappear on test, revealing that optimizers tend to add redundant rules that increase token usage.

By Jiawei Zhou, Bo Peng, Yuxuan Huang, Ming Zhang, Youbang Sun, Xiang Wang, Chaochao Lu
arXiv AI
Oct 28

M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents

M4Bench is a benchmark comprising 28 clinical derivation tasks across 15 families on MIMIC‑IV and eICU, designed to evaluate whether AI coding agents can reproduce complex procedural specifications for electronic health record (EHR) variable derivation. The study tests agents with and without clinician‑reviewed skill sets, finding that specification completeness drives most performance gains, while generic context or skill format contribute little. The benchmark includes tasks, skills, evaluation code, and sanitized evidence, all released as open‑source.

By Hannes Ill, Rafi Al Attrach, Rajna Fani, Ahram Han, Ryohei Yamamoto, Marlene Louisa Moerig, Dukyong Yoon, Leo Anthony Celi
arXiv AI
Oct 28

From Interaction to Capability: Task Advancement as State and Supervision for GUI Agents

The paper introduces Task Advancement Memory (TAM), a framework that tracks verified task progress to improve GUI agent performance. TAM defines a Task Advancement State that captures how progress changes remaining goals, enabling agents to build on earlier achievements across applications. After each task, TAM’s advancement supervision abstracts interactions that yielded verified local results, preserving reusable operations and reducing lifecycle tokens.

By Ziyang Chen, Qian Li, Chengwei Qin
arXiv AI
Oct 28

What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

The paper investigates how language models acquire knowledge over the course of pretraining by using point-in-time models trained on web snapshots from different years. It introduces a temporal variability score (TVS) that measures how consistently a knowledge item is represented across these snapshots. The study finds that items with high TVS improve more rapidly later in training and outperform low‑TVS items in most benchmark settings, especially on knowledge‑intensive tasks.

By Chang Su, Bin Wu, Emine Yilmaz, Yao Lu
arXiv AI
Oct 28

KPPax: preprocess differentiable kinetic ODE for machine learning in atmospheric chemistry

KPPax is a new kinetic preprocessor that converts atmospheric chemistry mechanisms written in the Kinetic PreProcessor (KPP) language into differentiable, just‑in‑time compiled, vectorisable code in JAX. It evaluates rate laws in log space to maintain differentiability and integrates stiff systems using implicit solvers with forward‑ and reverse‑mode adjoints from the diffrax library, enabling exact sensitivities without hand‑derived adjoint code. Demonstrations with the POLLU benchmark show that KPPax can predict which rate coefficients are identifiable and improve neural‑network corrections of emission inventories, reducing errors by up to a factor of five.

By Wenqing Peng, Michael Boy, Zhi-Song Liu
arXiv AI
Oct 28

Who Convinced Whom? Tracing How LLMs Change Their Minds in Multi-Agent Systems

The paper introduces MAD Bench, a 2,400-question benchmark for multi‑agent debate (MAD) that tests knowledge, reasoning, values, moral judgment, safety, and refusal. It evaluates three open‑weight models and finds that peer communication can alter individual decisions, with forced‑choice readouts before rationale generation more often matching the final answer in debate than in reflection. The study shows that stronger consensus can coexist with increased shared errors, suggesting the need for evaluation metrics that consider dependence and unanimous errors alongside accuracy.

By Arnesh Batra, Aniket Khandelwal, Jashn Khemani, Krish Thukral, Arush Gumber, Denis Federico Lim, Ujwal Kumar, Phan Xuan Tan
arXiv AI
Oct 28

Curriculum-Driven Deep Reinforcement Learning for Autonomous Far-Range Rocket Landing

The paper introduces a curriculum-driven deep reinforcement learning framework for autonomous far-range rocket landing. It uses a three-stage curriculum, critic warm‑up, rehearsal to avoid catastrophic forgetting, and a high‑dimensional observation vector with a full 3×3 rotation matrix. The redesigned reward function bounds per‑step rewards and accumulates progress, yielding a 99.98% success rate over 100,000 episodes, far surpassing the baseline.

By Atin Chowdhury, Piyush Joshi
arXiv AI
Oct 28

Multiple Leak Localization in Water Distribution Networks through Graph Neural Networks

The paper introduces a graph neural network (GNN) that can locate multiple simultaneous leaks in water distribution networks without predefining the number of leaks or exhaustively enumerating all possible leak combinations. It is tested on two benchmark networks—the 31-node Hanoi and the 92-node NET3—under varied demand, sparse sensor coverage, and different leak magnitudes. The GNN outperforms traditional classifiers such as multilayer perceptron, support vector machine, K-nearest neighbors, and random forest by up to 12% and 20% in accuracy on the Hanoi and NET3 networks, respectively, demonstrating scalable performance.

By Arvin Ajoodani, Sara Nazif, Pouria Ramazi