Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 31

Integrated Feature Selection for classification and Representation Learning with Group-Lasso Siamese Networks

The paper introduces a novel Siamese network framework that integrates Group‑Lasso for feature selection, aiming to identify compact feature subsets that preserve intrinsic data representations while maintaining high classification performance. Two variants are proposed—one using Euclidean distance and another employing a modified Jaccard similarity for real‑valued vectors—to better handle high‑dimensional sparse data. Experiments on 12 benchmark datasets with feature dimensions up to 7,070 demonstrate that the method consistently selects discriminative features and achieves competitive or superior classification results across eight classifiers, with embedding quality evaluated via UMAP and t‑SNE.

By Nisargo Ranjan Pal, Pradeep Singh
arXiv AI
Oct 31

Can LLMs Faithfully Solve Ontology Engineering Tasks? A Mechanistic Interpretability Study

The paper introduces MIOE, a causal‑abstraction benchmark designed to probe how large language models (LLMs) handle ontology engineering tasks such as entity typing and subsumption prediction. Using this benchmark, the authors evaluate several mechanistic interpretability methods—including SAE, DBM, and DAS—on Qwen3‑8B and Qwen3‑4B models, finding that distributed subspace interventions perform best but still mix with auxiliary class‑specific information. The study shows that while ontological predictions are accessible at a mechanistic level, current techniques do not yet isolate a clean, disentangled causal mechanism for ontological knowledge.

By Bohui Zhang, Albert Mero\~no-Pe\~nuela, Elena Simperl
arXiv AI
Oct 31

Co-Evolving Policy and Rubric via Capability Awareness for Open-Ended Generation

The paper introduces CoCA, a closed‑loop framework that jointly evolves a policy and its rubric by diagnosing each rubric criterion against the current policy’s capabilities. CoCA contrasts query‑only and rubric‑guided rollouts to identify mismatched difficulty, recalibrates criteria, and uses frontier‑based learning or self‑distillation to align the rubric with the policy’s evolving strengths. Experiments on various benchmarks and large language models show that CoCA consistently outperforms existing rubric‑based reinforcement learning methods and helps the policy internalize guidance‑revealed capabilities while preventing degradation.

By Chengsheng Zhang, Wenshen Xu, Yuchuan Wu, Zhiyuan Yu, Junfeng Fang, Tianlong Wang, Bingxue Qiu, Yongbin Li, Xiang Wang
arXiv AI
Oct 31

Deogen: Automatic Benchmark Generation for Reasoning over Policies

DeoGen is an automatic, scalable, and parametric benchmark generator for deontic reasoning, which involves applying policy rules to facts to determine verdicts. It autoformalizes policy rules into first‑order logic, builds decision trees whose paths become benchmark entries, and produces combinatorially many entries per tree depth. Evaluations show state‑of‑the‑art models achieve 89‑92% accuracy on verdict‑determining paths but only 22‑40% on paths lacking sufficient information, even when instructed to abstain.

By Sam Kouteili, William Liang, Keeran Murray, Adriana Zhou, Ramesh Viswanathan, Scott Shapiro, Ruzica Piskac
arXiv AI
Oct 31

TreasuryQA Bench: Can AI Agents Work an Archive of Scanned Financial Records?

TreasuryQA Bench is a new benchmark that tests AI agents on real-world tasks involving searching, reading, and extracting exact figures from scanned U.S. Treasury PDFs stored on a shared drive. The benchmark presents 70 questions over 1,435 scanned documents, requiring agents to locate the relevant file via Dropbox’s Model Context Protocol, OCR the necessary pages, and return a precise numeric answer. Results from ten agents show a wide performance gap, with the best agent solving 64.8% of tasks and the worst only 16.7%, and scan quality is shown to significantly impact success rates.

By Ritiz Tambi, Sahar Shayegan, Joshua Meyer, Sun Kim, Ali Khan, Victor Shih, Mehdi Jamei, Andi Partovi
arXiv AI
Oct 31

Certified Access: Test-Time Compute on Conservative Physics-Structured AI Memory

The paper introduces a theoretical and empirical framework for test‑time compute on a Latent Dynamics Model used as memory. It shows that mechanisms such as retries, confidence‑based escalation, and non‑local routing can be formally analyzed using a damped symplectic step of a learned Hamiltonian, yielding exact phase‑volume Jacobians, bounded energy changes, and latch‑transport laws. The authors verify a certificate stack on an analytic testbed, demonstrating how escape failures are resolved by a Lorentz squeeze and reach failures by a gated canonical translation, and evaluate scaling limits against a physics‑free baseline.

By Pratik Jawahar, Riccardo Maggioni, Jonas Petersen, Matteo Cozzi, Gian-Alessandro Lombardi, Marek Gajewski, Camilla Mazzoleni, Federico Martelli
arXiv AI
Oct 31

Beyond Similarity Search: Knowledge Graph Metadata Enrichment for Enterprise Retrieval-Augmented Generation

The paper investigates whether a knowledge graph can enhance retrieval for enterprise retrieval‑augmented generation (RAG) systems compared to traditional similarity search. Using a snapshot of SEC EDGAR filings, the authors built a graph with 430,257 nodes and 561,144 edges, then evaluated three architectures—text‑to‑SQL, dense‑vector RAG, and a two‑stage GraphRAG. GraphRAG outperformed the other methods, correctly answering 16 of 20 questions across all attempts, while the other approaches lagged behind, especially in structured aggregation and single‑document queries.

By Fakabbir Amin
arXiv AI
Oct 31

SAG-Sep: Sparse Augmented Graphs and Onion-Guided Search for Rounded Capacity Cut Separation

SAG‑Sep is a new separator for rounded capacity cuts (RCCs) in capacitated vehicle‑routing problems that uses a single encoder pass to score node membership across vehicle‑count levels, avoiding iterative predict‑and‑coarsen steps. It encodes the LP solution as a Sparse Augmented Graph (SAG) with flow‑informed multi‑hop edges, and leverages the nested, onion‑like structure of high‑value RCCs through a training‑free Onion‑Sweep decoder that ranks customers and emits the most violated prefixes. On the NeuralSEP benchmark with 1,000‑customer instances, SAG‑Sep with Onion‑Sweep cuts the average root gap by 17.7% over the best NeuralSEP baseline and achieves a roughly 30‑fold per‑iteration speedup.

By Haoran Liu, Guanyi Wang, Yu Yang
arXiv AI
Oct 31

Estimating Implicit Expected Log Density Ratios

The paper introduces the concept of an expected log density ratio (ELDR), which involves integrating log density ratios under a third measure. It demonstrates that many problems involving implicit densities are actually ELDR estimation tasks and establishes fundamental limitations for finite-sample ELDR estimation. The authors argue that standard density ratio estimation methods are unsuitable for ELDR and propose a simple modification to existing DRE techniques that improves ELDR estimation error in benchmarks.

By Avi Amalanshu, Chandler Squires, Yizhou Lu, Pradeep Ravikumar
arXiv AI
Oct 31

Scaling Lean Theorem Proving for Real-World Code Verification

The paper introduces EasyProver, a Lean theorem prover that uses recursive agents, isolated contexts, asynchronous search, parallel tactic probing, and an elastic elaboration service. It achieves near‑saturation results on several benchmark suites, proving 669/672 PutnamBench problems, all six IMO 2026 formalizations, 597/600 NTP4VC verification conditions, and all 189 VERINA tasks. Additionally, the authors present Aegis, a pipeline that transforms competitive programming problems into harder verification tasks by generating safe Rust solutions, translating them to Lean, and producing 2,045 Rust‑Lean program pairs with 11,533 candidate specifications, enabling new benchmarks for provers and reinforcement learning.

By Debargha Ganguly, Divya Raghunathan, Dejan Jovanovi\'c, Vikash Singh, Soonho Kong, Aaron Tomb, Aman Goel, Ferhat Erata, Sam Bayless
arXiv AI
Oct 31

Skeleton Coding: Program Structure as a Specification for Agents

The paper introduces Skeleton Coding, a method where developers jointly specify a program’s structure—its data types, modules, and functions—alongside behavioral tests. This approach allows the agent to first commit to a structure, narrowing the search space of possible programs, and then use tests to select the correct implementation. Empirical results across multiple models and benchmarks show that skeletons can dramatically reduce the number of required tests, improve accuracy, and lower implementation costs.

By Aditya Golatkar, Alessandro Achille, Renos Zabounidis, Vlad Sobal, Yangruibo Ding, Baishakhi Ray, Jonathan Buck, Guy Van den Broeck, Maryam Fazel, Pietro Perona, Wei Xia, Stefano Soatto
arXiv AI
Oct 31

EPSBench: Benchmarking LLM Agents for Earnings Forecasting

EPSBench is a benchmark that evaluates large language model agents on quarterly earnings forecasting by having them connect financial evidence to income‑statement line items for 149 firm‑quarters from 86 US companies. Agents read filings, earnings‑call transcripts, and news, estimate company‑specific drivers, and a shared accounting calculator converts these into GAAP diluted earnings per share (EPS). Six models were assessed, with three showing no significant difference from analyst consensus in point forecast error, while the others performed worse; nominal 80% prediction intervals contained reported EPS only 37–70% of the time, largely due to low operating‑income estimates.

By Yanbin Yin, Chuyu Xu, Liqin Ye, Jing Peng, Chao Zhang
arXiv AI
Oct 31

A Case Study of a Multi-Agent Kernel-Optimization Loop That Learns a New AI Accelerator

The article describes a case study where a large language model (LLM) agent loop was used to optimize three Gemma‑4‑12B decode kernels for the FuriosaAI RNGD NPU, a sparsely documented Rust DSL accelerator. Over 12 days on a shared device, the loop built its own target knowledge, performed microbenchmarks, recorded compiler legality rules, and iteratively refined kernel designs through role‑based search and acceptance tests. The resulting kernels achieved a 9.293× geometric mean speedup over the official baseline, each within 1.25× of the theoretical cycle minimum for weight movement.

By Byonggon Chun
arXiv AI
Oct 31

Benchmarking decision models against generative large language models on medical multiple-choice questions

The study compared 13 decision models with 15 generative large language model configurations on 1,273 four‑option MedQA questions. Decision models achieved 67.71%–89.08% accuracy, while generative models ranged from 78.32%–96.86%. Mercury Decide offered the lowest cost per 1,000 questions and Clef Omni the fastest median request time, but decision models still showed higher error rates for high‑confidence predictions and varied probability repeatability.

By Oscar Freyer, Daniel Glauert, Moritz Hofmann, Max Ostermann, Stephen Gilbert, Jakob Nikolas Kather
arXiv AI
Oct 31

How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0

The study compares deterministic static application security testing (SAST) with AI-powered coding agents and security harnesses on the VulnBench 2.0 benchmark, which contains 108 curated vulnerabilities across 20 open‑source web applications. Nine productized configurations—including Snyk Code, Claude Code, Codex Security, and DeepSec—were evaluated under a blind protocol, with Snyk Code achieving 82.4% macro recall and Claude Code 79.3%. Together, the AI configurations identified 101 of the 108 vulnerabilities, surpassing any single tool, and also uncovered seven additional real vulnerabilities not present in the reference set, highlighting the complementary strengths of AI harnesses and deterministic SAST.

By Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth, Brian Clark
arXiv AI
Oct 31

SatOps-Decision: Low-Latency Discriminative Decisions for Onboard Satellite Communication Operations

SatOps-Decision is a low‑latency decision‑making method for satellite communication operations that uses a locally adapted Laya encoder with a native decision head to select operational answers in a single forward pass. It is the first application of Jev‑style discriminative decision models to satellite operations, and its accompanying benchmark is the first dedicated to this approach, featuring authored operational contracts, controlled state changes, and independently checked labels. In evaluations, SatOps-Decision achieves a median latency of 90.3 ms per decision and a mean raw accuracy of 85.3 %, outperforming baseline models such as Qwen3.5‑0.8B and OneJev1.6‑0.8B.

By Lifan Mei, Yuxuan Li, Jiaxian Tu, Shenghe Xu
arXiv AI
Oct 31

Benchmarking Graph Neural Networks for AC Power Flow: The Role of Inductive Biases and Physical Constraints

The paper presents a unified benchmark of graph neural network (GNN) architectures for predicting AC power flow (ACPF) on IEEE 14‑bus and 118‑bus systems. It isolates the effects of message‑passing mechanisms and inductive biases by using identical preprocessing, training procedures, and comparable parameter budgets. The study finds that anisotropic message‑passing, explicit edge features for line impedances, and global context aggregation yield the best trade‑off between predictive accuracy and physical consistency, while two or three message‑passing layers are generally optimal under a fixed parameter budget.

By Federico Clerici, Ferran Bohigas-Daranas, Hamid Latif-Mart\'inez, Eduardo Prieto-Araujo, Oriol Gomis-Bellmunt, Pere Barlet-Ros
arXiv AI
Oct 31

Towards Generalizing Pretrained All-Atom Representations to Unified Binder Design

PairFuse is a lightweight, parameter‑efficient method that fuses pretrained all‑atom representations into a geometric latent diffusion model for unified binder design. By adding layerwise adapters, it enriches the diffusion model’s molecular block features and incorporates pairwise structural information into the denoiser. Experiments on antibody, peptide, and small‑molecule binder‑design benchmarks show that PairFuse outperforms both the base diffusion model and existing representation‑fusion baselines.

By Maojiang Su, Jiale Chen, Qijie Zhu, Shang Wu, Jiayi Wang, Jihai Zhao, Guo Ye, Yibo Wen, Han Liu
arXiv AI
Oct 31

Revisiting Over-refusal in Safety-aligned Large Language Models: A Representation and Optimization Perspective

The paper investigates why safety‑aligned large language models still over‑refuse harmless queries, attributing the issue to overlapping representations in deeper layers and imbalanced gradient coupling that favors refusal. It proposes GOAL, a framework that orthogonalizes over‑refusal gradients against a low‑rank refusal subspace and adds contrastive learning to sharpen harmful‑vs‑over‑refusal representation separation. Experiments show GOAL reduces over‑refusal while maintaining safety and general performance.

By Chao Chen, Pengyang Shao, Wei Qin, Shijie Hao, Fei Shen, Richang Hong
arXiv AI
Oct 31

ThinkJev: One Thought, Every Budget for Calibrated Typed Decisions

ThinkJev introduces a hybrid approach that lets a language model decide whether to answer directly or to think first, using the same underlying model for both fast and slow responses. By reading the model’s probability distribution over option letters after any number of thought tokens, the system can provide a calibrated decision under any thinking budget without retraining. Experiments with Qwen3.5-27B show that the method achieves high accuracy while reducing the number of thought tokens needed, and it performs competitively on the JevBench benchmark.

By Xiaolu Ding, Kai Yang, Dongdong Liu