Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,065 stories · RSS feed

arXiv AI

Jul 23

MOF-Sleuth: Tool-Grounded Reward Alignment for Explainable Fine-Grained MOF CIF Auditing

arXiv:2607. 19935v1 Announce Type: new Abstract: Large metal-organic framework (MOF) databases support simulation, screening, and machine learning through crystallographic information files (CIFs).

By Yu Liu, Zhiwei Yang, Diandian Guo, Kun Peng, Fangfang Yuan, Cong Cao, Chaozhuo Li, Zhiyuan Ma, Yanbing Liu, Guobin Zhao

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jul 23

PRIME-SVR: Physics-infoRmed Implicit Multi-Echo Slice-to-Volume Reconstruction for Fetal T2 mapping

arXiv:2607. 20136v1 Announce Type: cross Abstract: Slice-to-volume reconstruction (SVR) is the standard method for obtaining high-resolution (HR) 3D fetal brain volumes from motion-corrupted 2D MRI slice stacks acquired in multiple orientations.

By Busra Bulut, Maik Dannecker, Thomas Sanchez, Sara Neves Silva, Steven Jia, Jean-Baptiste Ledoux, Leo Pomar, Joanna Sichitiu, Yvan Gomez, Meriam Koob, Vincent Dunet, Maria Deprez, Guillaume Auzias, Francois Rousseau, Jana Hutter, Daniel Rueckert, Meritxell Bach Cuadra

benchmarks

arXiv AI

Jul 23

Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis

arXiv:2607. 20216v1 Announce Type: cross Abstract: Malware analysis demands rapid interpretation of complex detonation reports spanning filesystem, network, and process behaviours.

By Adel ElZemity, Shujun Li, Budi Arief

llms agents benchmarks safety

arXiv Machine Learning

Jul 23

Predictive single cell foundation model for gene regulation and aging with privacy-preserving tabular learning

arXiv:2607. 19400v1 Announce Type: new Abstract: Pre-trained foundation models (FMs) have begun transforming single-cell genomics, but scaling them raises privacy concerns.

By Jiayuan Ding, Jianhui Lin, Ziyang Miao, Nils Mechtel, Shiyu Jiang, Yixin Wang, Zhaoyu Fang, Jorge D. Martin-Rufino, Chen Weng, Reuben Saunders, Weize Xu, Jonathan S. Weissman, Min Li, Jiliang Tang, Wei Ouyang, Yuancheng Ryan Lu, Xiaojie Qiu

agents benchmarks safety

arXiv AI

Jul 23

DQAOA-GPT: AI-Accelerated Distributed Quantum Optimization for Combinatorial Problems

arXiv:2607. 20225v1 Announce Type: cross Abstract: While combinatorial optimization problems are central to many scientific and engineering applications, their solution remains challenging due to exponentially large search spaces.

By Seongmin Kim, Abhinav Rijal, Yuri Alexeev, Nora Bauer, Martin Roetteler, Mina Yoon, George Siopsis, In-Saeng Suh

llms benchmarks

arXiv AI

Jul 23

Reference-Free Evaluation of Reasoning in Open-Ended Question Answering

arXiv:2607. 19678v1 Announce Type: cross Abstract: AI-generated answers in high-stakes domains are often fluent but difficult to verify, especially when they contain multi-step reasoning rather than a single final answer.

By Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull, Gregory E Dean, Maria Liakata

llms nlp benchmarks

arXiv AI

Jul 23

Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Review

arXiv:2507. 10142v2 Announce Type: replace Abstract: Multi-Agent Reinforcement Learning (MARL) has achieved strong performance in simulated benchmarks, yet real deployments often violate the assumptions under which algorithms are designed and evaluated.

By Siyi Hu, Mohamad A Hady, Jianglin Qiao, Jimmy Cao, Mahardhika Pratama, Ryszard Kowalczyk

agents reinforcement-learning benchmarks

arXiv AI

Jul 23

In-the-Flow Agentic System Optimization for Effective Planning and Tool Use

arXiv:2510. 05592v2 Announce Type: replace Abstract: Outcome-driven reinforcement learning has advanced reasoning in large language models (LLMs), but prevailing tool-augmented approaches train a single, monolithic policy that interleaves thoughts and tool calls under full context; this scales poorly with long horizons and diverse tools and generalizes weakly to new scenarios.

By Zhuofeng Li, Haoxiang Zhang, Seungju Han, Sheng Liu, Jianwen Xie, Yu Zhang, Yejin Choi, James Zou, Pan Lu

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 23

An Auditable Policy-Simulation Framework for Student Dropout in Intervention-Free Data

arXiv:2604. 08874v3 Announce Type: replace-cross Abstract: This study proposes a temporal modeling framework with a counterfactual policy-simulation layer for student dropout in higher education, using LMS engagement data and administrative withdrawal records.

By Rafael da Silva, Jeff Eicher, Gregory Longo

benchmarks

arXiv AI

Jul 23

NeuroSymActive: Differentiable Neural-Symbolic Reasoning with Active Exploration for Knowledge Graph Question Answering

arXiv:2602. 15353v3 Announce Type: replace-cross Abstract: Large pretrained language models and neural reasoning systems have advanced many natural language tasks, yet they remain challenged by knowledge-intensive queries that require precise, structured multi-hop inference.

By Rong Fu, Yang Li, Zeyu Zhang, Jiekai Wu, Yaohua Liu, Shuaishuai Cao, Yangchen Zeng, Yuhang Zhang, Xiaojing Du, Simon Fong

llms rag nlp benchmarks

arXiv AI

Jul 23

Geometry-Guided Constraint Learning for LLM Safety Classification

arXiv:2607. 19366v1 Announce Type: new Abstract: Safety as Polytope (SaP) learns linear half-space constraints in LLM hidden space but requires per-category tuning of the constraint count K.

By Fumiaki Uehara, Koo Imai, Masato Tsutsumi, Keigo Kansa, Sora Usui, Yuki Kobiyama

llms benchmarks safety

arXiv AI

Jul 23

OpenEvoShield: Dual Non-Stationary Continual Defense for Open-World Multi-Agent System Attacks

arXiv:2607. 19351v1 Announce Type: new Abstract: LLM-based multi-agent systems (LLM-MAS) are increasingly deployed in safety-critical applications, where adversaries inject malicious instructions through inter-agent communication to propagate harmful behaviors.

By Litian Zhang, Chaozhuo Li, Yuting Zhang, Zejian Chen, Bingyu Yan, Qiwei Ye

llms agents benchmarks safety

arXiv AI

Jul 23

G-MAD: A Game-Based Data Generation Framework for Multi-View RGB-T Aerial Object Detection

arXiv:2607. 19942v1 Announce Type: cross Abstract: This work introduces G-MAD, an open-source framework that uses Arma3 to generate synchronized multi-view RGB-T data for aerial object detection.

By Yechan Kim, JongHyun Park, Dongho Yoon, Namhoon Jung, Moongu Jeon

computer-vision multimodal benchmarks safety

arXiv AI

Jul 23

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

arXiv:2607. 19539v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures increase model capacity without proportionally increasing computation cost and have become a key building block for scaling large language models (LLMs) to trillion-parameter regimes.

By Minyu Cui, Anna Wingkvist, Morgan Ericsson

llms benchmarks

arXiv AI

Jul 23

Recovering Clinical Utility Under Differential Privacy: Empirical Validation of Adaptive Federated Aggregation on Heterogeneous Cardiovascular Datasets

arXiv:2607. 19403v1 Announce Type: cross Abstract: Validating federated learning frameworks on real clinical data is an essential step between proof-of-concept demonstrations in controlled synthetic environments and deployment in real multicenter healthcare settings.

By Rodrigo Tertulino, Laercio Alencar, Ricardo Almeida

benchmarks safety

arXiv AI

Jul 23

Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models

arXiv:2607. 19847v1 Announce Type: cross Abstract: Predicting missing cell values in tabular data is a fundamental problem in data cleaning.

By Yurong Liu, Yeye He, Haoyu Dong, Junjie Xing, Shi Han, Dongmei Zhang, Surajit Chaudhuri

llms benchmarks

arXiv AI

Jul 23

SynPre-FL: Synthetic data-driven pretraining integrated Federated Learning training framework

arXiv:2607. 19524v1 Announce Type: cross Abstract: Federated learning (FL) offers a promising approach to privacy-preserving clinical risk prediction, but its deployment remains limited by restricted data sharing, client heterogeneity, class imbalance, and the lack of realistic tabular electronic health record (EHR) benchmarks.

By Akarsh K Nair, Muhammad Arifur Rahman, Nicholas Shopland, Andy Burton, Jun He, Yuan Shen, David Baldwin, Emma O'Dowd, Amna Burzic, Mufti Mahmud, David J. Brown

diffusion benchmarks safety

arXiv Machine Learning

Jul 23

CityGuard: Graph-Aware Private Descriptors for Bias-Resilient Identity Search Across Urban Cameras

arXiv:2602. 18047v4 Announce Type: replace-cross Abstract: City-scale person re-identification across distributed cameras must handle severe appearance changes from viewpoint, occlusion, and domain shift while complying with data protection rules that prevent sharing raw imagery.

By Rong Fu, Yibo Meng, Jia Yee Tan, Rui Lu, Jiekai Wu, Simon Fong

llms rag efficiency benchmarks safety

arXiv AI

Jul 23

SCPP: A Unified Python Library for Soft Clustering

arXiv:2607. 19620v1 Announce Type: cross Abstract: In this paper, we present SCPP (Soft Clustering Python Package), an open-source Python framework for soft clustering.

By Kiyan Rezaee, Morteza Ziabakhsh, Artin Bahrampour, Seyed Mohammad Ghoreishi, Asal Khaje, Ali Sajedifar, Manny Chalak, Ava Zerafatangiz, Sadegh Eskandari

benchmarks

arXiv AI

Jul 23

Hybrid LSTM-Graph Neural Framework for Robust Financial Fraud Detection and Adversarial Resilience

arXiv:2607. 19350v1 Announce Type: new Abstract: Financial institutions face significant challenges in detecting sophisticated money laundering patterns, such as smurfing and layering, due to extreme data imbalance (0.

By Mariam Zakaria Moussa Ali

benchmarks safety

Get the daily briefing