Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,065 stories · RSS feed

arXiv AI

Jul 22

Supra Cognitive Modes: A Routed Architecture for Agent Memory

arXiv:2607. 19096v1 Announce Type: new Abstract: Agent-memory workloads mix direct factual lookup, relation-chain and current-state reasoning, and broad synthesis over long histories.

By Joshua Tobkin, David Yang

rag agents benchmarks safety

arXiv Machine Learning

Jul 22

Mark, Don't Erase: Token Inoculation for Dual-Use Knowledge in LLMs

arXiv:2607. 18639v1 Announce Type: new Abstract: Safety interventions on dual-use knowledge typically choose between destroying hazardous content (e.

By Seunghyun Lee, Dongyoon Han, Sangdoo Yun

llms fine-tuning benchmarks safety

arXiv Machine Learning

Jul 22

S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning

arXiv:2607. 19232v1 Announce Type: new Abstract: Hierarchical Reinforcement Learning (HRL) intends to separate strategic planning from primitive execution.

By Kshitij Kumar Srivastava, Kshitij Jerath

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 22

Finite-Agent Stochastic Differential Games on Large Graphs: II. Graph-Based Architectures

arXiv:2509. 12484v2 Announce Type: replace Abstract: We propose a novel neural network architecture, called Non-Trainable Modification (NTM), for computing Nash equilibria in stochastic differential games (SDGs) on graphs.

By Ruimeng Hu, Jihao Long, Haosheng Zhou

rag agents robotics benchmarks safety

arXiv AI

Jul 22

Cross-Dialect Generalization Without Retraining: Benchmarks and Evaluation of Schema-Derived Constrained Decoding for MLIR

arXiv:2607. 18254v1 Announce Type: new Abstract: Multi-Level Intermediate Representation (MLIR) underlies modern ML compiler infrastructure (TensorFlow, JAX/StableHLO, PyTorch Inductor, IREE), yet appears only in trace amounts in code-LM pretraining corpora.

By Plawan Kumar Rath

fine-tuning benchmarks

arXiv AI

Jul 22

Structured Synthetic Reasoning Data for Arithmetic Fine-Tuning of Small Language Models

arXiv:2607. 18266v1 Announce Type: new Abstract: Small language models are attractive for local deployment, but they often struggle with multi-step arithmetic reasoning.

By Jake O'Grady, Effirul Ramlan

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 22

Uncertainty Quantification for AI-Driven Crash Simulation Surrogates: A Comparative Study of Monte Carlo Dropout and Deep Ensemble on Open-Source Bumper Beam Benchmark

arXiv:2607. 18294v1 Announce Type: new Abstract: Machine learning surrogate models are increasingly being explored in engineering product development to augment simulation-driven design, offering near-instantaneous predictions that complement computationally expensive high-fidelity analyses.

By Sudeep Chavare

benchmarks safety

arXiv AI

Jul 22

Fly0: Persistent Metric Anchoring for Zero-Shot Aerial Vision-Language Navigation

arXiv:2602. 15875v2 Announce Type: replace-cross Abstract: Current Visual-Language Navigation (VLN) methodologies face a trade-off between semantic understanding and control precision.

By Zhenxing Xu, Yihong Lu, Weidong Bao, Zhengqiu Zhu, Jingxuan Zhou, Zhichuang Wang, Ji Wang, Lihua Liu, Wei He

llms multimodal benchmarks

arXiv Machine Learning

Jul 22

Is EEG-to-Text Feasible in Real-World Scenarios? An In-Depth Analysis Using a Neuropsychology-Inspired Benchmark

arXiv:2607. 18749v1 Announce Type: new Abstract: Translating brain signals into text could restore communication for people with severe paralysis, yet practically usable systems to date rely on invasive electrocorticography (ECoG).

By Zihan Zhang (Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology), Yu Bao (Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, Shanghai Innovation Institute), Xiao Ding (Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology), Tianyi Jiang (State Key Laboratory for Novel Software Technology, Nanjing University), Kai Xiong (Zhongguancun Laboratory)

benchmarks

arXiv AI

Jul 22

BatchDAG: LLM-Planned Execution Graphs for Scalable Ad-Hoc Analysis Over Enterprise Data

arXiv:2607. 18241v1 Announce Type: new Abstract: Large language models (LLMs) excel at analyzing individual documents but break down on exhaustive, cross-entity analytical questions over enterprise-scale datasets due to context overflow, loss of per-entity attribution, and linear latency from sequential tool calls.

By Anupreet Walia

llms agents benchmarks safety data-engineering

arXiv Machine Learning

Jul 22

Local Label-Informed Feature Transfer for Generating Ground-Truth Medical Images: A Comparison of GAN- and Diffusion-Based Approaches

arXiv:2607. 18882v1 Announce Type: cross Abstract: Validating Explainable Artificial Intelligence (XAI) methods in medical imaging requires ground-truth data with known locations of informative features.

By Rick Wilming, Irem Ozseker, Luca Matteo Cornils, Ahc\`ene Boubekki, Benedict Clark, Danny Panknin, Stefan Haufe

diffusion benchmarks

arXiv AI

Jul 22

When Does Machine Learning Beat Value Sorting? A Three-Dataset Diagnostic of Exposure-Weighted Shipment Prioritization

arXiv:2607. 18573v1 Announce Type: new Abstract: Delay-risk models are usually judged by predictive accuracy.

By Jize Li

benchmarks safety

arXiv AI

Jul 22

SysAdmin: Measuring Instrumental Power-Seeking in Frontier AI

arXiv:2607. 18239v1 Announce Type: new Abstract: Power-seeking defined as behaviors where AI systems acquire resources, evade oversight, or resist termination beyond task requirements is identified as a key driver of Loss of Control (LoC) risk.

By Mana Azarm, Qiyao Wei, Rahul Nambiar

llms agents benchmarks safety

arXiv AI

Jul 22

MILP-Evo: Closed-Loop Fully Automatic Design of MILP Solvers

arXiv:2607. 18252v1 Announce Type: new Abstract: Machine learning methods have shown that data-driven policies can accelerate mixed-integer linear programming (MILP) solvers, but many such approaches remain difficult to inspect, adapt, and deploy because the learned policy is represented as an external predictor or other opaque model.

By Jinbiao Nie, Kewei Feng, Xiaoyuan Zhang, Shan Yin, Zizhuo Wang, Bin Dong

llms benchmarks

arXiv Machine Learning

Jul 22

PertReason: A Knowledge-Grounded Benchmark and Framework for Cell-State-Conditioned Mechanistic Reasoning of Perturbation Effects

arXiv:2607. 18777v1 Announce Type: new Abstract: Evaluating machine learning in scientific domains requires separating correct predictions from correct reasons under realistic distribution shifts.

By Dongkwan Kim, Yiming Gao, Yining Yang, Yang Shen

llms benchmarks

arXiv AI

Jul 22

SechKAN: Kolmogorov-Arnold Networks with Hyperbolic Secant Functions

arXiv:2607. 18290v1 Announce Type: cross Abstract: In recent years, Kolmogorov-Arnold Networks (KANs) have attracted increasing attention due to their effectiveness in machine learning and scientific computing tasks, offering a new paradigm for neural network design.

By Hoang-Thang Ta

computer-vision benchmarks

arXiv AI

Jul 22

BioSecBench-Refusal: A paired metric for performance and alignment in agentic biosecurity risk assessment

arXiv:2607. 05462v2 Announce Type: replace-cross Abstract: As AI agents are incorporated into life science workflows, the capabilities that speed discovery might also enable misuse.

By Edwin H. Wintermute, Harmon Bhasin, Christina M. Agapakis, Dianzhuo Wang, Evan Seeyave, Arjun Banerjee, Daniel Fulop, Matthew C. Watson, Adam J. Meyer, Sandrine Boissel, Jens H. Kuhn, Rishi Jain, Noah D. Taylor, Helena Shomar, Patrick M. Boyle, Kenny Workman

agents benchmarks safety

arXiv AI

Jul 22

MUX: Continuous Reasoning via Multiplexed Tokens

arXiv:2607. 18264v1 Announce Type: new Abstract: Language models solve complex problems by articulating intermediate reasoning steps in natural language.

By Ayhan Suleymanzade, Halil Alperen Gozeten, Michael Bronstein, \.Ismail \.Ilkan Ceylan, Jinwoo Kim

llms diffusion efficiency benchmarks

arXiv AI

Jul 22

Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

arXiv:2607. 18438v1 Announce Type: cross Abstract: Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs' ability to complete an assortment of tasks from distinct domains in a single prompt.

By Liam Swayne

llms multimodal benchmarks

arXiv AI

Jul 22

Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models

arXiv:2607. 18280v1 Announce Type: cross Abstract: Large language models (LLMs) are often compressed through static parameter pruning or dynamic token-level computation, yet aggressive sparsification can trigger rapid performance degradation beyond an essential sparsity boundary.

By Chao Han, Haozhe Hu, Xiaoyu Shen

llms nlp efficiency benchmarks

Get the daily briefing