Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv AI

Jun 16

From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents

arXiv:2606. 04990v2 Announce Type: replace-cross Abstract: Large language model (LLM)-based agents are evolving from passive text generators into autonomous systems capable of planning, tool use, retrieval, memory access, environmental interaction, and multi-agent collaboration.

By Yiqi Wang, Jiaqi Zhang, Taotao Cai, Zirui Liu, Qingqiang Sun, Zequn Sun, Zhangkai Wu, Manqing Dong, Mingkai Zhang, Xuefei Yin, Yanming Zhu

llms agents benchmarks safety

arXiv AI

Jun 16

PVminerLLM2: Improving Structured Extraction of Patient Voice via Preference Optimization

arXiv:2606. 16074v1 Announce Type: cross Abstract: Motivation: Patient-generated text contains critical information on patients' lived experiences, social context, and care engagement, but remains largely unstructured, limiting its use in patient-centered outcomes research.

By Samah Fodeh, Linhai Ma, Ganesh Puthiaraju, Srivani Talakokkul, Afshan Khan, Elyas Irankhah, Sreeraj Ramachandran, Ashley Hagaman, Sarah Lowe, Aimee Roundtree

llms fine-tuning benchmarks

arXiv AI

Jun 16

Is My Vision-Language Data in Your AI? Membership Inference Test (MINT) Demo 2

arXiv:2606. 14748v1 Announce Type: cross Abstract: We present the Membership Inference Test (MINT) Demo 2, a framework designed to improve transparency in machine learning training processes.

By Daniel DeAlcala, Gonzalo Mancera, Julian Fierrez, Aythami Morales, Ruben Tolosana, Ruben Vera-Rodriguez

llms multimodal benchmarks

arXiv AI

Jun 16

Tool-IQA: Augmenting Image Quality Assessment with Simple Tools

arXiv:2606. 16082v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have been increasingly adopted for Image Quality Assessment (IQA).

By Guanyi Qin, Junjie Zhang, Chunming He, Yibing Fu, Jie Liang, Tianhe Wu, Lei Zhang

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 16

Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models

arXiv:2602. 16793v2 Announce Type: replace Abstract: In the past year, custom and unreleased math reasoning models reached gold medal performance on the International Mathematical Olympiad (IMO).

By Xingyu Dang, Rohit Agarwal, Rodrigo Porto, Anirudh Goyal, Liam H Fowl, Sanjeev Arora

llms benchmarks

arXiv AI

Jun 16

Mojo: A Promising Tool for Scalable Financial AI Efficiency

arXiv:2606. 16059v1 Announce Type: cross Abstract: For thirty years, quantitative finance has paid a costly two-language tax: models researched in Python are rewritten in C++ for production, often introducing numerical discrepancies.

By Henry Han

llms benchmarks

arXiv AI

Jun 16

Do Safety Monitors Stay Reliable After an Update? Benchmarking and Predicting Activation-Monitor Staleness

arXiv:2606. 15980v1 Announce Type: cross Abstract: Activation monitors-lightweight probes trained on a language model's internal representations-are an increasingly common layer in deployment safety stacks.

By Evan Duan

llms fine-tuning efficiency benchmarks safety

arXiv AI

Jun 16

Control-Plane Placement Shapes Forgetting: An Architectural Study of Agent Memory Across Thirteen System Configurations

arXiv:2606. 15903v1 Announce Type: cross Abstract: Where an LLM sits in an agent memory pipeline -- between the recall plane that retrieves stored facts (extensively benchmarked) and the control plane that mutates them via supersede, release, purge (largely untested) -- shapes which forgetting failure modes the system recovers.

By Dongxu Yang

llms agents benchmarks safety

arXiv Machine Learning

Jun 16

Evaluating Local Explainability Metrics for Machine Learning Models on Tabular Data

arXiv:2605. 27618v2 Announce Type: replace Abstract: Despite the wide use of explainability techniques to attempt to understand the behavior of Artificial Intelligence (AI), the generated explanations may not always be reliable.

By Tom\'as Pereira, Jo\~ao Vitorino, Eva Maia, Isabel Pra\c{c}a

benchmarks

arXiv AI

Jun 16

SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills

arXiv:2606. 15899v1 Announce Type: cross Abstract: Open-source LLM agent ecosystems are growing rapidly, yet the security of community-contributed skills - modular tool definitions that extend agent capabilities - remains largely unvetted.

By Ismail Hossain, Sai Puppala, Md Jahangir Alam, Tanzim Ahad, Sajedul Talukder

llms agents benchmarks

arXiv Machine Learning

Jun 16

MacrOData: New Benchmarks of Thousands of Datasets for Tabular Outlier Detection

arXiv:2602. 09329v3 Announce Type: replace Abstract: Quality benchmarks are essential for fairly and accurately tracking scientific progress and enabling practitioners to make informed methodological choices.

By Xueying Ding, Simon Kl\"uttermann, Haomin Wen, Yilong Chen, Leman Akoglu

benchmarks

arXiv Machine Learning

Jun 16

Analytic Torsion and Spectral Gap Capture Persistent-Laplacian Performance

arXiv:2606. 16990v1 Announce Type: new Abstract: While persistent Laplacians (PL) offer a richer geometric representation of data than persistent homology, utilizing their full eigenspectrum for learning tasks is often hampered by high dimensionality and the ``varying length'' problem across different filtration scales.

By Jernej Grlj, Aaron D. Lauda

benchmarks

arXiv AI

Jun 16

Deep Residual Injection for Full-Spectrum Forensic Signal Perception in Multimodal Large Language Models

arXiv:2606. 15880v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have been increasingly adopted in forensics for their robust semantic understanding.

By Kaiqing Lin, Zhiyuan Yan, Ruoxin Chen, Ke-Yue Zhang, Yue Zhou, Caiyong Piao, Bin Li, Taiping Yao, Bo Wang, Youchang Xiao, Shouhong Ding

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 16

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

arXiv:2606. 15768v1 Announce Type: cross Abstract: Vision-Language-Action models (VLAs) leverage large-scale vision-language pretraining for semantic robot control, but often lack explicit foresight into how robot actions change the scene.

By Jialei Chen, Kai Wang, Kang Chen, Shuaihang Chen, Feng Gao, Wenhao Tang, Zhiyuan Li, Weilin Liu, Zhuyu Yao, Boxun Li, Yuanbo Xu, Chao Yu

diffusion robotics multimodal benchmarks

arXiv AI

Jun 16

Imperfect Visual Verification for Code Edition : A Case Study on TikZ

arXiv:2606. 15693v1 Announce Type: cross Abstract: LLMs have significantly advanced code generation, enabling the synthesis of functional programs.

By Charly Reux (UR, INSA Rennes, DiverSe), Mathieu Acher (CNRS, IUF, IRISA, UR, DiverSe), Djamel Eddine Khelladi (DiverSe, UR, CNRS, IRISA), Cl\'ement Quinton (SPIRALS, CNRS), Olivier Barais (UR, IRISA, DiverSe)

llms benchmarks

arXiv AI

Jun 16

MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs

arXiv:2606. 15694v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in understanding complex multimodal content.

By Hangling Xie

llms nlp multimodal benchmarks

arXiv AI

Jun 16

LLMs on Tabular Data with Limited Semantics: Evidence from Industrial Car Retrofit Prediction

arXiv:2606. 15314v1 Announce Type: cross Abstract: Industrial retrofit planning depends on structured operational data rather than free text: planners must estimate whether a newly registered prototype will require a retrofit, which retrofit package it will need, and how long the work will take.

By Aina Vila Pons, Ioannis Tzachristas, Constantinos Antoniou

llms rag benchmarks safety

arXiv AI

Jun 16

The Missing Knowledge Layer in Cognitive Architectures for AI Agents

arXiv:2604. 11364v2 Announce Type: replace Abstract: The two most influential cognitive architecture frameworks for AI agents, CoALA [21] and JEPA [12], both lack an explicit Knowledge layer with its own persistence semantics.

By Micha\"el Roynard

llms rag agents benchmarks

arXiv AI

Jun 16

Evolutionary Dynamics of Cooperation in Next-Generation LLM Agent Systems: A Cross-Provider Empirical Extension

arXiv:2605. 29874v2 Announce Type: replace-cross Abstract: Do next-generation LLM agents inherit the cooperative biases documented in their predecessors, or does scale and provider diversity reshape equilibrium behaviour in competitive multi-agent settings?

By Francisco Le\'on Z\'u\~niga Bol\'ivar (Instituci\'on Universitaria Colegio Mayor del Cauca)

llms agents benchmarks safety

arXiv AI

Jun 16

CAP: Towards PPG Universal Representation Learning with Patient-level Supervision

arXiv:2606. 15284v1 Announce Type: cross Abstract: Photoplethysmography (PPG) plays a central role in wearable health monitoring and clinical decision support.

By Chenyang He, Xinyi Shao, Shun Huang, Bosong Huang, Daoqiang Zhang, Ming Jing, Cheng Ding

multimodal benchmarks safety

Get the daily briefing