Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,452 stories · RSS feed

arXiv Machine Learning

Jun 18

FORGE: Foundational Optimization Representations from Graph Embeddings

arXiv:2508. 20330v5 Announce Type: replace Abstract: Combinatorial optimization problems are ubiquitous in science and engineering.

By Zohair Shafi, Serdar Kadioglu

rag fine-tuning efficiency benchmarks

arXiv AI

Jun 18

Surrogate Benchmarks for Model Merging Optimization

arXiv:2509. 02555v2 Announce Type: replace-cross Abstract: Model merging techniques aim to integrate the abilities of multiple models into a single model.

By Rio Akizuki, Yuya Kudo, Nozomu Yoshinari, Yoichi Hirose, Toshiyuki Nishimoto, Kento Uchida, Shinichi Shirakawa

llms benchmarks

arXiv Machine Learning

Jun 18

Adaptive Speech-to-Spike Encoding for Spiking Neural Networks

arXiv:2606. 19039v1 Announce Type: cross Abstract: The mismatch between continuous acoustic signals and discrete event-driven processing remains a fundamental bottleneck for neuromorphic speech processing.

By Taharim Rahman Anon, Jakaria Islam Emon

benchmarks safety

arXiv AI

Jun 18

SciRisk-Bench: A Risk-Dimension-Aware Benchmark for AI4Science Safety

arXiv:2606. 18936v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly embedded in AI for Science (AI4Science) workflows, from scientific question answering and literature analysis to laboratory planning and autonomous discovery.

By Linghao Feng, Yinqian Sun, Dongqi Liang, Sicheng Shen, Chenfei Yan, Yuxuan Peng, Yilin Zhao, Haibo Tong, Kai Li, FeiFei Zhao, Yi Zeng

llms agents nlp benchmarks safety

arXiv AI

Jun 18

A Controlled Benchmark of Quantum-Latent GAN Augmentation for Brain MRI

arXiv:2606. 18970v1 Announce Type: cross Abstract: Medical image classification is often constrained by limited labeled data, motivating generative augmentation; recently, quantum generative models have been proposed for this purpose, frequently reporting accuracy gains.

By Syed Mujtaba Haider, Silvia Figini

diffusion computer-vision benchmarks

arXiv AI

Jun 18

Fully Geometric Multi-Hop Reasoning on Knowledge Graphs with Transitive Relations

arXiv:2505. 12369v2 Announce Type: replace Abstract: Multi-hop logical reasoning on knowledge graphs requires faithfully mapping the logical semantics to latent space.

By Fernando Zhapa-Camacho, Robert Hoehndorf

rag diffusion benchmarks safety

arXiv AI

Jun 18

SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents

arXiv:2606. 18733v1 Announce Type: cross Abstract: Realistic coding-agent benchmarks often replay public GitHub issues and pull requests, making them vulnerable to overlap with model pretraining, fine-tuning, synthetic-data generation, or benchmark-driven model selection.

By Qiao Zhao, JianYing Qu, Jun Zhang, Yehua Yang, Hanwen Du, Zhongkai Sun

agents fine-tuning benchmarks

arXiv AI

Jun 18

RedactionBench

arXiv:2606. 18782v1 Announce Type: cross Abstract: Large Language Models are increasingly applied to sensitive domains that require redaction of personally identifiable information (PII).

By Sean Brynj\'olfsson, Shashvat Jayakrishnan, Esha Sali, Diptanshu Purwar, Madhav Aggarwal

llms agents nlp benchmarks safety

arXiv AI

Jun 18

Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

arXiv:2606. 19266v1 Announce Type: cross Abstract: The development of large language models (LLMs) has led to an increased focus on their adaptation to specialized domains and languages, yet the effectiveness of domain adaptation strategies remains unclear.

By Ikram Belmadani, Oumaima El Khettari, Carlos Ramisch, Frederic Bechet, Richard Dufour, Benoit Favre

llms fine-tuning benchmarks

arXiv AI

Jun 18

Hilbert-Geo: Solving Solid Geometric Problems by Neural-Symbolic Reasoning

arXiv:2605. 16385v3 Announce Type: replace-cross Abstract: Geometric problem solving, as a typical multimodal reasoning problem, has attracted much attention and made great progress recently, however most of works focus on plane geometry while usually fail in solid geometry due to 3D spatial diagrams and complex reasoning.

By Ruoran Xu, Haoyu Cheng, Bin Dong, Qiufeng Wang

llms multimodal benchmarks

arXiv AI

Jun 18

WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks

arXiv:2604. 06367v2 Announce Type: replace-cross Abstract: Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries.

By Guruprasad Viswanathan Ramesh, Asmit Nayak, Basieem Siddique, Kassem Fawaz

llms agents multimodal benchmarks safety

arXiv AI

Jun 18

TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering

arXiv:2604. 28076v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have advanced Table Question Answering, where most queries can be answered by extracting information or simple aggregation.

By An-Yang Ji, Jun-Peng Jiang, De-Chuan Zhan, Han-Jia Ye

llms agents nlp benchmarks

arXiv AI

Jun 18

Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Survey

arXiv:2602. 15851v2 Announce Type: replace-cross Abstract: Applications of narrative theories using large language models (LLMs) deliver promising methods in automatic story generation and understanding tasks.

By David Y. Liu, Aditya Joshi, Paul Dawson

llms nlp benchmarks

arXiv AI

Jun 18

Multi-Modal Hyper-Graph Fusion for Low-Light Crowd Counting

arXiv:2606. 18566v1 Announce Type: cross Abstract: Crowd counting is a fundamental task in computer vision.

By Hao-Yuan Ma, Li Zhang, Yushi Qiu, Jie Gao, Yan Zhang, Bangjun Wang

computer-vision multimodal benchmarks

arXiv AI

Jun 18

SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval

arXiv:2606. 18801v1 Announce Type: cross Abstract: With the rapid expansion of massive multilingual corpora, Multilingual Information Retrieval (MLIR) has emerged as a critical technology for global information access.

By Youngjoon Jang, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim

rag benchmarks safety

arXiv Machine Learning

Jun 18

Formalizing and Mitigating Structural Distortion in LLM Attention for Graph Reasoning

arXiv:2606. 15633v2 Announce Type: replace Abstract: Large Language Models (LLMs) have shown promise for reasoning over Text-Attributed Graphs (TAGs).

By Donald Loveland, Puja Trivedi, Ari Weinstein, Edward W Huang, Danai Koutra

llms rag benchmarks

arXiv AI

Jun 18

SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents

arXiv:2606. 18356v1 Announce Type: cross Abstract: Tool-using language-model agents introduce security failures that go beyond unsafe text: they can disclose protected objects, write persistent memory, send messages, modify databases, or trigger harmful code and tool effects.

By Yuchuan Tian, Mengyu Zheng, Haocheng Mei, Ye Yuan, Chao Xu, Xinghao Chen, Hanting Chen, Yu Wang

llms agents benchmarks safety

arXiv Machine Learning

Jun 18

Task-Restricted Symmetries in Recurrent Weight Space

arXiv:2606. 18457v1 Announce Type: new Abstract: Recurrent networks can contain substantial functional redundancy in weight space: changing a recurrent matrix may leave the input-output rollout nearly unchanged on a task distribution, while similar-scale changes can destroy the same behavior.

By Simon Dr\"ager

agents benchmarks

arXiv AI

Jun 18

LLM Compression by Block Removal with Constrained Binary Optimization

arXiv:2602. 00161v2 Announce Type: replace-cross Abstract: In this paper, we formulate the compression of large language models (LLMs) by optimally deleting transformer blocks (``block removal'') as a constrained binary optimization (CBO) problem that can be mapped to a physical system (Ising glass), whose energies are a strong proxy for downstream model performance.

By David Jansen, Roman Rausch, Ali Hashemi, David Montero, Rom\'an Or\'us

llms benchmarks

arXiv AI

Jun 18

Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion

arXiv:2606. 18469v1 Announce Type: cross Abstract: Neuroscientific research has revealed that the brain encodes complex behaviors by leveraging structured, low-dimensional manifolds and dynamically fusing multiple sources of information through adaptive gating mechanisms.

By Somjit Nath, Jackson J Cone, Derek Nowrouzezahrai, Samira Ebrahimi Kahou

rag reinforcement-learning benchmarks

Get the daily briefing