Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv Machine Learning
Sep 11

CausalArena: Benchmarking Causal Discovery in the Foundation Model Era

CausalArena is a new benchmark designed to evaluate causal discovery methods in the era of foundation models. It unifies synthetic structural causal models (SCMs), semantically grounded SCMs, and formula‑grounded SCMs, while also including real‑world datasets for external validation. Experiments show that performance rankings vary widely across different SCM families and protocols, indicating that strong results on one benchmark do not necessarily transfer to others.

By Zi-Rong Li, Si-Yang Liu, Tian-Zuo Wang, Han-Jia Ye
arXiv Machine Learning
Sep 11

Optimizing AI Inference Across the Deployment Stack

The paper argues that AI deployment performance depends on interactions among compression, compiler transformations, and serving policies rather than just model architecture. It introduces a three‑layer taxonomy—model‑level techniques, compiler transformations, and system policies—and frames deployment as a constrained multi‑objective optimization problem over accuracy, latency, throughput, memory footprint, and energy. The authors propose an evidence protocol for comparable benchmarking and synthesize data from edge and data‑center platforms to show that cross‑layer interactions drive deployment outcomes, concluding with a constraint‑aware selection procedure and open research problems.

By Tejinder Singh, John Pflueger, Jeebak Mitra, Robert Lincourt, Mitchell Markow, Bhavesh A. Patel
arXiv Machine Learning
Sep 11

EVTradeMatch: A Mobility-Aware Multi-Objective Matching Framework for EV--EV Energy Trading

EVTradeMatch is a mobility-aware, multi-objective matching framework that coordinates peer-to-peer energy trading between electric vehicles (EVs). It uses a prediction-guided score for charging-node suitability and formulates the matching problem as a mixed-integer linear program, solved via a tailored NSGA-II algorithm. Experiments show significant gains in transferred energy, charging-node suitability, and matching coverage compared to existing proximity- and auction-based methods.

By Md. Mahfujur Rahman, Alistair Barros, Raja Jurdak, Darshika Koggalahewa
arXiv Machine Learning
Sep 11

Adaptive Diffusion Freezing: Privacy-preserving Diffusion Models Against Membership Inference Attacks

Adaptive Diffusion Freezing (ADF) is a new privacy‑preserving framework for diffusion models that protects against membership inference attacks (MIAs). It uses cross‑timestep adaptive freezing training, where a mask matrix controls which data subsets participate at each diffusion timestep, reducing over‑memorization and aligning model behavior for member and non‑member samples. A pretraining‑based risk‑aware freezing policy estimates MIA risk and suppresses high‑risk subset‑timestep pairs, achieving a superior privacy‑utility‑efficiency trade‑off across multiple datasets.

By Jialu Guo, Xiao Han, Junjie Wu
arXiv AI
Sep 11

OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows

OpenDiscoveryTrace is a public dataset of 558 complete AI scientific agent trajectories that records the reasoning process—thoughts, tool calls, observations, errors, revision triggers, and confidence—across 124 scientific tasks in drug discovery, materials science, genomics, and literature analysis. The dataset includes seven models (three frontier models and four open‑weight models) and 60 live‑retrieval variants, providing a balanced view of performance and error patterns. Pilot analysis shows that process traces reveal behavioral differences invisible to output‑only evaluation, such as differing error rates and types among frontier models.

By Aayam Bansal, Keertan Balaji
arXiv AI
Sep 11

Multi-Agent Agentic Graph Learning via Structural Signatures

The paper introduces Multi-Agent Agentic Graph Learning (MAAGL), a framework that partitions a graph into communities and assigns a dedicated agent to each community for specialized reasoning. MAAGL addresses two key challenges in existing agentic graph learning: it preserves permutation invariance by summarizing structural evidence with a dynamic structural signature, and it controls context size by filtering semantic evidence to the top‑k relevant nodes. Experiments on four benchmark datasets demonstrate that MAAGL outperforms state‑of‑the‑art agentic graph learning methods.

By Liang Qu, Jianxin Li, Hua Wang
arXiv AI
Sep 11

Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection

The paper critiques the common reinforcement‑learning approach of sampling tool subsets when the full set of tools is enumerable, showing that sampling leads to degraded policy estimates and increased reward sparsity in genomic reasoning tasks. It proposes Full‑Group Policy Optimization (FGPO), which evaluates every tool subset and precomputes rewards in a table, thereby eliminating the need for frozen‑reasoner calls during training. Experiments across five frozen reasoners and three genomic benchmarks demonstrate that FGPO consistently outperforms GRPO, improving average scores by 6.75 points and reducing the number of invoked tools per question.

By Haoyue Liu, Xiaoyu Ma, Ye Chen, Zhichao Wang, Xiaoying Tang
arXiv AI
Sep 11

DiffLUT-Net: Differentiable Training of FPGA LUT Networks with Learnable Connectivity

DiffLUT-Net is an FPGA-native neural‑network architecture that uses six‑input lookup tables (LUTs) trained from scratch. The method jointly learns each LUT’s 64 truth‑table entries and the source connections to its six input ports through a differentiable LUT function relaxation and hardware source selection. After training, the learned truth tables and connections are discretized, unused logic is pruned, and the network is exported as synthesizable Verilog, achieving favorable accuracy‑resource trade‑offs across five benchmarks.

By Jiaqi Ye, Xinrui Gong, Jingcun Wang, Olga Kondrateva, Bing Li, Grace Li Zhang
arXiv AI
Sep 11

Quantifying Logical Consistency in Transformers via Query-Key Alignment

The paper introduces a lightweight evaluation method for logical reasoning in transformer-based language models, using query-key alignments within attention heads to compute a QK-score. This single forward-pass technique identifies valid versus invalid inferences and is validated across multiple reasoning benchmarks, showing robustness to distractors and deeper reasoning. Experiments span models from 1.5B to 70B parameters, demonstrating scalability.

By Eduard Tulchinskii, Anastasia Voznyuk, Laida Kushnareva, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov
arXiv AI
Sep 11

CS-Guard: Benchmarking LLM Guardrails for Code Generation Security

CS-Guard is a new benchmark that systematically evaluates guardrails for code generation security, covering 1,000 malware-generation prompts, 7 jailbreak attacks, and a novel fictional scenario attack (FSA) for text-to-code generation, as well as 331 code prompts for code-to-code generation. The study empirically tests nine guardrails across seven large language models, finding that many guardrails fail to prevent malicious code generation, with attack success rates reaching about 50% for text-to-code and up to nearly 100% for code-to-code and FSA scenarios. CS-Guard introduces a modular three-layer guardrail taxonomy and releases its benchmark and data to support future research.

By Jinyang Li, Mingyu Guo, Hung X. Nguyen
arXiv AI
Sep 11

Grounded Continuation: A Linear-Time Runtime Verifier for LLM Conversations

Grounded Continuation introduces a runtime verifier that classifies each utterance in an LLM conversation into one of eight epistemic operations and uses a symbolic engine to maintain a dependency map of claims and their supports. The verifier checks whether a new continuation is grounded by walking this map, a linear-time process that requires no additional LLM calls. On benchmarks such as ReviseQA and MemoryAgentBench, the verifier improves single-hop accuracy for several QA models, even enabling a 7B model to outperform GPT‑4o when guided by the verifier.

By Qisong He, Jinwei Hu, Xinmiao Huang, Changshun Wu, Yi Dong, Xiaowei Huang
arXiv Machine Learning
Sep 11

A Multi-Stage Rule-Chaining Framework for Compositional and Interpretable Cognitive Reasoning

The paper introduces a multi-stage rule‑chaining framework for the Abstraction and Reasoning Corpus (ARC), aiming to model cognitive generalization by inferring abstract rules from few examples. It combines three solvers—a deterministic rule discovery module, a pattern‑composition engine, and a structural abstraction layer—executed sequentially in a fallback hierarchy that reuses earlier reasoning traces to improve interpretability and generalization. The system achieved over 95% accuracy on ARC tasks, demonstrating strong performance across deterministic, compositional, and abstract categories.

By Deblina Kar
arXiv Machine Learning
Sep 11

The Platonic brain bridge hypothesis: human brain networks as an architectural prior for omni models

The paper introduces the Platonic brain bridge hypothesis, asserting that omni models—capable of jointly processing video, audio, and text—naturally develop brain‑like representations, and that this relationship is bidirectional. Experiments show that seven omni models exhibit stable brain‑like representations across participants, and their internal states outperform others on the Algonauts 2025 out‑of‑distribution leaderboard. Building on this, the authors propose three brain‑to‑model methods: Brain‑MoE assigns a brain‑pretrained expert to each of seven cortical networks, improving benchmark accuracy; Brain‑AVQA generates video‑based questions using the most responsive brain network, outperforming shuffled mappings; and Brain‑Scope uses sparse autoencoders to pinpoint a small subset of networks whose removal weakens brain prediction, demonstrating that human brain networks can serve as a practical architectural prior for omni models.

By Pengfei Zhang, Biao Tian, Xiangang Li, Li Liu
arXiv Computer Vision
Sep 11

TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents

TeleOCR is a unified framework for document parsing that tackles challenges in both decoupled and end-to-end Vision‑Language Models. It introduces deformation‑aware learning to handle geometric distortions, an adaptive sampling mechanism for complex layouts, and a content‑structure decoupled strategy to model formula grammars and table structures. The approach achieves state‑of‑the‑art results on multiple benchmarks, including top placement in the ICDAR 2026 Sci‑ImageMiner Challenge.

By Peng Cai, Zhaofan Zou, Shifa Liu, Yikun Wang, Jiawei Tang, Kaicheng Yang, Meng Tong, MingKun Jiang, Zhongjiang He, Hao Sun
arXiv Computer Vision
Sep 11

FastMap: Real-Time Semantic Map Completion via Bitwise Masked Modeling

FastMap is a lightweight two‑stage framework that completes top‑down categorical semantic maps in real time. It first compresses map patches into compact bitwise tokens using a lookup‑free BitVAE, producing a 0.41 GB model that is 3.7× smaller than prior masked‑modeling baselines. A Masked AutoEncoder transformer then reconstructs missing tokens in a single forward pass (0.011 s per map), and an object‑aware masking strategy enables goal‑conditioned generation without extra inference cost. On the Gibson benchmark, FastMap attains 34.10 % mIoU and 45.84 % semantic Success Rate, more than doubling the previous best, and achieves an 83.8 % Success Rate on downstream ObjectNav, the highest among compared navigation methods.

By Yijie Deng, Shuaihang Yuan, Congcong Wen, Hao Huang, Anthony Tzes
arXiv Computer Vision
Sep 11

SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data

SegCol is a new dataset and benchmark for semantic segmentation of colon fold edges and surgical instruments in colonoscopy images, derived from the EndoMapper dataset. It offers manually annotated pixel‑level masks for three instrument classes and thin fold‑edge structures across temporally consistent image sequences, and serves as the basis for the SegCol Challenge within the EndoVis Challenge at MICCAI 2024. The study evaluates supervised segmentation and annotation‑efficient active learning, analyzes various segmentation metrics under structural perturbations, and highlights how metric behavior depends on target structure, underscoring the need for carefully selected evaluation protocols in endoscopic segmentation.

By Xinwei Ju, Rema Daher, Razvan Caramalau, Baoru Huang, Danail Stoyanov, Francisco Vasconcelos
arXiv Machine Learning
Sep 11

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

LILA (Latent-Informed Layer Analysis) introduces a calibration‑free method for structured pruning of large language models by scoring neuron importance using the Kolmogorov–Smirnov distance between singular value distributions of full and neuron‑ablated feed‑forward network weight matrices. The approach requires no training, calibration data, or auxiliary networks, and outperforms existing methods such as PruneNet and SliceGPT on LLaMA‑2‑7B and Phi‑2 at various sparsity levels. After a single epoch of LoRA fine‑tuning, LILA matches heavily calibrated baselines, and a Neural Tangent Kernel analysis provides theoretical support for its spectral importance criterion. Additionally, LILA can dynamically allocate sparsity budgets, achieving state‑of‑the‑art generative preservation and revealing architectural bottlenecks at higher compression.

By Sankar Behera, Dhruv Singh, Anshika Agnihotri, Raj Kumar Choudhary, Satyadev Ahlawat, Yamuna Prasad
arXiv Computation and Language
Sep 11

Xiaomi-CocktailASR-1 Technical Report

Xiaomi-CocktailASR-1 is an LLM-based end‑to‑end target‑speaker ASR architecture that uses reference speech as voiceprint prompts to transcribe a target speaker directly from mixed audio, eliminating the need for speech separation. It retains competitive single‑speaker performance comparable to mainstream ASR models and introduces a negative sample rejection capability, outputting empty text when the target speaker is absent. The system also supports a Chain‑of‑Thought reasoning mode and achieves state‑of‑the‑art performance on synthetic and real‑world multispeaker benchmarks, effectively addressing the cocktail party problem with a unified architecture.

By Yiru Zhang, Hang Su, Lichun Fan, Ying Zeng, Chang Liu, Yifeng Wang, Yuquan Liang, Tao Li, Lian Li, Wenhao Yang, Jian Luan, Cong Zou, Heng Qu
arXiv AI
Sep 11

Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

The paper presents an end‑to‑end pipeline for translating natural language planning descriptions into PDDL problem instances using large language models. It incorporates multiple checks—syntactic parsing, planner success, domain conformance, an LLM critic, and iterative repair—to ensure faithfulness to the original task. Experiments on Planetarium, AutoPlanBench, and curated PDDL~2.1 problems reveal that operational success can diverge from benchmark‑reference reconstruction, and that structured repair improves outcomes while PDDL~2.1 remains challenging for reference reconstruction.

By Joana Rosa, Pedro Santos, Valdemar Oliveira, Rom\~ao Silva, L. Miguel Silveira, Bruno Martins
arXiv Machine Learning
Sep 11

BiHDTrans: binary hyperdimensional transformer for efficient multivariate time series classification

BiHDTrans is a neurosymbolic binary hyperdimensional transformer that merges self‑attention with hyperdimensional computing to classify multivariate time series efficiently. It surpasses existing HD models by at least 14.47% and binary transformers by 6.67% on average, while an FPGA‑accelerated implementation reduces inference latency 39.4× compared to state‑of‑the‑art binary transformers. Even with a 64% reduction in hyperspace dimensionality, BiHDTrans remains competitive, achieving 1–2% higher accuracy with 4.4× smaller model size and nearly 50% lower latency than the full‑dimensional baseline.

By Jingtao Zhang, Yi Liu, Qi Shen, Changhong Wang