Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,067 stories · RSS feed

arXiv AI

Jul 9

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

arXiv:2607. 06799v1 Announce Type: cross Abstract: Evaluating uncertainty in AI-generated SQL queries requires estimating whether a query is correct, where correct means it executes to the same result as a human-written reference.

By Robert Richardson

llms fine-tuning benchmarks data-engineering

arXiv AI

Jul 9

At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics

arXiv:2607. 06639v1 Announce Type: cross Abstract: On modular arithmetic, a network's embedding keeps compressing for tens of thousands of steps after it has already generalized.

By Truong Xuan Khanh

llms rag benchmarks safety

arXiv Machine Learning

Jul 9

Generalist Vision-Language Models for Fast Radio Burst detection: a zero-shot benchmark against a specialized detector

arXiv:2607. 07382v1 Announce Type: new Abstract: Fast Radio Bursts (FRBs) are millisecond-duration radio transients whose automated detection increasingly relies on highly specialized deep learning models.

By Raiff H. Santos, Amilcar R. Queiroz, Tharcisyo S. S. Duarte, K. E. L. de Farias, Rafael A. Batista

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 9

Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval

arXiv:2605. 11325v3 Announce Type: replace-cross Abstract: Current LLM memory benchmarks evaluate answer quality rather than retrieval accuracy.

By Jeffrey Flynt

llms rag benchmarks

arXiv AI

Jul 9

Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks

arXiv:2607. 07189v1 Announce Type: new Abstract: Vision-language models (VLMs) and agentic AI have shown strong performance on semantic visual tasks, but it remains unclear whether they can handle the physics and inverse problems that underlie computational imaging.

By Ethan Chung, Chuanjun Zheng, Jasper Tan, Jingxi Li, Haopeng Zhang, Huaijin Chen

llms agents multimodal benchmarks

arXiv Machine Learning

Jul 9

Dual Attention Heads for Personalized Federated Learning in ECG Classification

arXiv:2607. 06653v1 Announce Type: new Abstract: Federated learning (FL) enables collaborative model training across institutions without sharing sensitive patient data.

By Kien Le, Joseph Lindley, Quoc Bao Phan, Tuy Tan Nguyen

llms benchmarks

arXiv Machine Learning

Jul 9

Robust Human-AI Complementarity under Uncertainty

arXiv:2607. 06656v1 Announce Type: new Abstract: Machine learning models are often intended to augment rather than replace human decision makers, by providing information that is complementary to human judgement.

By Yewon Byun, Bryan Wilder

benchmarks

arXiv AI

Jul 9

A Study of Commonsense Reasoning over Visual Object Properties

arXiv:2508. 10956v3 Announce Type: replace-cross Abstract: Inspired by human categorization, visual reasoning about object properties, such as physical attributes and functions, involves identifying and recognizing low-level details and higher-level abstractions.

By Abhishek Kolari, Mohammadhossein Khojasteh, Yifan Jiang, Floris den Hengst, Filip Ilievski

llms nlp multimodal benchmarks

arXiv AI

Jul 9

Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluations

arXiv:2607. 07229v1 Announce Type: new Abstract: Prior work has shown that chain-of-thought (CoT) reasoning is often unfaithful: a model's stated reasoning does not reliably reflect the process that produced its output.

By Silvia Santano

benchmarks safety

arXiv AI

Jul 9

Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safety

arXiv:2607. 07695v1 Announce Type: new Abstract: We introduce institutional red-teaming, an evaluation methodology for testing deployment rules in multi-agent AI: hold the agents, objectives, and task state fixed, vary only one rule, and attribute the resulting change in collective behavior to that rule.

By Yujiao Chen

llms agents benchmarks safety

arXiv AI

Jul 9

Latency-Aware Bid Acceptance under Operational Feasibility: A Public Benchmark with Hindsight Ceilings

arXiv:2607. 07343v1 Announce Type: cross Abstract: Online truckload bid acceptance is a closed-loop stochastic decision problem in which a carrier or broker must, in real time, accept or reject a tendered load subject to operational feasibility, fleet repositioning costs, and opportunity cost against future demand.

By Aswin Chandrasekaran

benchmarks

arXiv AI

Jul 9

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

arXiv:2607. 07508v1 Announce Type: cross Abstract: Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs).

By Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong

llms agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jul 9

Creating Power Distribution Network Layouts Using Generative Adversarial Networks and Image-Based Representations

arXiv:2607. 06622v1 Announce Type: cross Abstract: Utilities increasingly rely on planning and operational tools to cope with the increased penetrations of distributed energy resources, yet the lack of realistic, openly available datasets remains a major barrier for benchmarking and comparison.

By Juan Manuel Garcia-Perez, Carlos Mateo

diffusion benchmarks safety

arXiv AI

Jul 9

AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning

arXiv:2607. 07033v1 Announce Type: cross Abstract: Large vision-language models incur substantial inference costs because high-resolution inputs introduce thousands of visual tokens, many of which are redundant for a given query.

By Kyuan Oh, Bumsoo Kim

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jul 9

LEMUR 2: Unlocking Neural Network Diversity for AI

arXiv:2607. 06839v1 Announce Type: new Abstract: Existing NAS benchmarks (e.

By Tolgay Atinc Uzun, Waleed Khalid, Saif U Din, Sai Revanth Mulukuledu, Akashdeep Singh, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Yashkumar Rajeshbhai Lukhi, Muhammad A. Hussain, Krunal Jesani, Usha Shrestha, Yash Mittal, Roman Kochnev, Pritam Kadam, Mohsin Ikram, Harsh R. Moradiya, Alice Arslanian, Dmitry Ignatov, Radu Timofte

llms rag diffusion fine-tuning multimodal benchmarks

arXiv AI

Jul 9

Do Counterfactually Fair Image Classifiers Satisfy Group Fairness? -- A Theoretical and Empirical Study

arXiv:2607. 06603v1 Announce Type: cross Abstract: The notion of algorithmic fairness has been actively explored from various aspects of fairness, such as counterfactual fairness (CF) and group fairness (GF).

By Sangwon Jung, Sumin Yu, Sanghyuk Chun, Taesup Moon

computer-vision efficiency benchmarks safety

arXiv AI

Jul 9

Self-Supervised Pretraining Improves Cross-Site and Cross-Scale Robustness of Point Cloud Leaf-Wood Segmentation

arXiv:2607. 06948v1 Announce Type: cross Abstract: The accuracy of existing leaf-wood segmentation methods for tree point clouds varies across forest types and sites.

By Heeju Mun, Tackang Yang, Yunsoo Nam, Changhyun Choi

computer-vision fine-tuning benchmarks

arXiv AI

Jul 9

RWGBench: Evaluating Scholarly Positioning in Related Work Generation

arXiv:2606. 24894v2 Announce Type: replace-cross Abstract: Large language models have shown strong fluency in scientific writing, yet the evaluation of related work generation (RWG) remains limited.

By Anzhe Xie, Weihang Su, Jiaxin Mao, Yiqun Liu, Shaoping Ma, Qingyao Ai

llms nlp benchmarks

arXiv Machine Learning

Jul 9

The Key to Going Linear: Analysis-Driven Transformer Linearization

arXiv:2607. 07706v1 Announce Type: new Abstract: The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference.

By Anna Kuzina, Paul N. Whatmough, Babak Ehteshami Bejnordi

llms benchmarks

arXiv AI

Jul 9

AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

arXiv:2607. 06624v1 Announce Type: new Abstract: We present AgentLens, a production-assessed benchmark for interactive code agents.

By Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, Sergey Nikolenko

llms agents benchmarks

Get the daily briefing