Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,789 stories · RSS feed

arXiv AI

Jun 24

Toward Self-Evolution-Ready Workflow Harnesses: A Reversible Migration Path and Convertibility Taxonomy for Expert LLM Pipelines

arXiv:2606. 24598v1 Announce Type: cross Abstract: While expert-validated "LLM + script" workflows deliver significant value, they remain static: they encode hard-won domain knowledge yet fail to adapt execution based on feedback.

By Yimo Lin, Zhen Zhang, Yibin Li

llms agents benchmarks

arXiv AI

Jun 24

Multimedia and Visual Analytics in the Agentic Era

arXiv:2504. 06138v3 Announce Type: replace-cross Abstract: Professional users need tools to help them gain actionable insights from large multimedia collections.

By Marcel Worring, Jan Zah\'alka, Stef van den Elzen, Maximilian T. Fischer, Daniel A. Keim

agents computer-vision benchmarks

arXiv Machine Learning

Jun 24

RE4: Transformation-aware Imitation of Object Interactions Using Manipulation Modes

arXiv:2606. 24403v1 Announce Type: cross Abstract: Object interaction tasks have been a focus of advances in imitation learning.

By Arsh Chawla, Rahul Shome

diffusion computer-vision robotics benchmarks safety

arXiv Machine Learning

Jun 24

Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective

arXiv:2512. 10244v2 Announce Type: replace-cross Abstract: Semi-supervised few-shot learning (SSFSL) resembles real-world applications such as auto-annotation, as it aims to learn a model from a few labeled and abundant unlabeled task-specific examples to annotate the unlabeled ones.

By Tian Liu, Anwesha Basu, James Caverlee, Shu Kong

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 24

You Don't Need to Run Every Eval

arXiv:2606. 24020v1 Announce Type: new Abstract: A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare design choices, and select the checkpoint for the release.

By Yuchen Zeng, Dimitris Papailiopoulos

benchmarks

arXiv AI

Jun 24

DeepBD: A Grounded Agentic Workflow for Variant Prioritization and Diagnosis of Genetic Birth Defects

arXiv:2606. 24779v1 Announce Type: cross Abstract: Birth defects are a major cause of fetal loss, neonatal morbidity and long-term disability.

By Shiyu Li, Ziqi Yan, Zhihao Wu, Jielong Lu, Weiran Liao, Jiajun Yu, Genjie Li, Zeyu Chu, Jiajun Bu, Haishuai Wang

llms rag agents benchmarks

arXiv AI

Jun 24

AI-PAVE-Br: Leveraging Large Language Models for Enhanced Product Attribute Value Extraction through a Golden Set Approach

arXiv:2606. 24655v1 Announce Type: cross Abstract: The explosive growth and complexity of product data within the dynamic Brazilian e-commerce landscape demand robust and specialized methods for structured information extraction.

By Murilo Gazzola, Hugo Gobato Souto, Samuel Silva, J\'ulia Schubert Peixoto, Felipe Siqueira, Andr\'e Luis Pedroso de Morais, Caio Gomes

llms nlp benchmarks

arXiv Machine Learning

Jun 24

A Hybrid, Multi-Layered Pipeline for Phishing and Threat Classification: Independently Validated URL and NLP Engines with a Calibrated Multi-Channel Fusion Stage

arXiv:2606. 21690v2 Announce Type: replace-cross Abstract: Phishing is a multi-modal threat.

By Saifelden M. Ismail, Aser O. Ibrahim, Omar A. Mahmoud

nlp multimodal benchmarks

arXiv Machine Learning

Jun 24

Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making

arXiv:2508. 16420v3 Announce Type: replace Abstract: Target-conditioned sequence models provide a simple interface for controllable offline decision making, but the requested target return can be an unreliable control signal, especially when the target return lies in underrepresented regions of the dataset.

By Yue Pei, Hongming Zhang, Chao Gao, Martin M\"uller, Yingying Zhang, Mengxiao Zhu, Hao Sheng, Ziliang Chen, Liang Lin, Haogang Zhu

llms benchmarks safety

arXiv AI

Jun 24

What Does ODRL Mean? A Cross-Level Ontological Grounding of Permissions, Prohibitions, and Duties in UFO-L

arXiv:2606. 24344v1 Announce Type: cross Abstract: ODRL policy evaluators produce verdicts, but say nothing about the normative positions a policy brings into existence, the authority structures those positions presuppose, or who holds the power to declare a norm violated.

By Daham M. Mustafa, Christoph Lange, Giancarlo Guizzardi, Diego Collarana, Christoph Quix, Stefan Decker

benchmarks

arXiv AI

Jun 24

Fix Initial Programs and Iteratively Refine Repair Instructions Toward Non-Elimination Multi-Turn Program Correction

arXiv:2604. 23989v2 Announce Type: replace-cross Abstract: Recent work on large language models (LLMs) has emphasized the importance of scaling inference compute.

By Yuto Tanaka, Issei Sato

llms benchmarks

arXiv AI

Jun 24

HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs

arXiv:2606. 23238v2 Announce Type: replace Abstract: Logical reasoning is essential for reliable AI, yet existing benchmarks are largely first-order-logic-centric, focusing on object-level deduction over fixed predicates.

By Yucheng Wu, Jundong Xu, Mingzhen Ju, Yue Yu, Chenpeng Wang, Haoxuan Li, Liangming Pan

llms benchmarks

arXiv AI

Jun 24

ASALT: Adaptive State Alignment for Lateral Transfer in Multi-agent Reinforcement Learning

arXiv:2606. 24601v1 Announce Type: new Abstract: Multi-agent reinforcement learning (MARL) addresses the problem of training multiple agents that pursue collaborative, competitive, or mixed objectives.

By Anurag Akula, Satheesh K. Perepu, Abhishek Sarkar, Kaushik Dey

rag agents reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 24

Neuromorphic Speech Enhancement with Dual-Branch Spiking Neural Networks

arXiv:2606. 23761v1 Announce Type: cross Abstract: Spiking neural network (SNN)-based neuromorphic speech enhancement has emerged as a promising paradigm due to its energy efficiency, yet it still underperforms classical artificial neural network (ANN)-based approaches owing to binary activations and the lack of well-designed network architectures.

By Taiyu Meng, Wenbin Jiang, Haoyi Zhang, Yuhan Zhou, Haibing Yin

benchmarks

arXiv AI

Jun 24

RetiSEM: Generalising Causal Models for Fragmented Biomedical Data

arXiv:2606. 24488v1 Announce Type: cross Abstract: Learning causal models from fragmented biomedical data is challenging because clinical, molecular, and imaging variables are often incomplete or not jointly observed.

By Inam Ullah, Imran Razzak, Shoaib Jameel

multimodal benchmarks

arXiv AI

Jun 24

On the Stability of Prompt Ranking in Large Language Model Evaluation

arXiv:2606. 24381v1 Announce Type: cross Abstract: Prompt-based interaction has become a dominant paradigm for using large language models (LLMs), where multiple candidate prompts are evaluated and the top-ranked one is selected for downstream use.

By Shaoshuai Du, Penghao Liang, Yixian Shen, Chuanqi Shi, Hang Zhang, Lun Wang

llms benchmarks

arXiv AI

Jun 24

Graph Alignment for Benchmarking Graph Neural Networks and Learning Positional Encodings

arXiv:2505. 13087v2 Announce Type: replace-cross Abstract: We propose a novel benchmarking methodology for graph neural networks (GNNs) based on the graph alignment problem, a combinatorial optimization task that generalizes graph isomorphism by aligning two unlabeled graphs to maximize overlapping edges.

By Adrien Lagesse, Marc Lelarge

llms rag benchmarks safety

arXiv Machine Learning

Jun 24

Exact Schur-Sylvester Dimensionality Reductions for Non-Smooth Stochastic Complexity and Manifold Sampling

arXiv:2606. 23867v1 Announce Type: new Abstract: The exact computation of the Normalized Maximum Likelihood (NML) codelength for regular non-smooth estimators (e.

By Trenton Lau, Gary P. T. Choi

benchmarks

arXiv AI

Jun 24

Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAs

arXiv:2606. 23938v1 Announce Type: new Abstract: Driving VLA models incorporating Chain-of-Thought (CoT) reasoning are attractive because they leverage pretrained VLM representations and expose intermediate decisions in natural language, yet current rationales often lack the step-by-step decision semantics needed to keep the rationale causally connected to the planned motion.

By Xiangbo Gao, Xiukun Huang, Boyu Lu, Junge Zhang, Mengjie Mao, Jiachen Li, Wei Xiong, Zhengzhong Tu

fine-tuning multimodal benchmarks safety

arXiv AI

Jun 24

FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representation

arXiv:2507. 16696v3 Announce Type: replace-cross Abstract: Industrial signal analysis is hindered by severe data heterogeneity, which we characterize as the M5 problem.

By Pingyi Fan, Anbai Jiang, Shuwei Zhang, Xinhu Zheng, Zhiqiang Lv, Bing Han, Wenrui Liang, Junjie Li, Wei-Qiang Zhang, Yanmin Qian, Xie Chen, Jia Liu

efficiency multimodal benchmarks

Get the daily briefing