Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,067 stories · RSS feed

arXiv AI

Jul 9

Reliable and Developer-Aligned Evaluation of Agents for Software Engineering

arXiv:2607. 06713v1 Announce Type: cross Abstract: Large language models are rapidly moving towards closing the development cycle, transitioning from simple assistive companions to autonomous contributors deeply embedded into collaborative development environments.

By Razvan Mihai Popescu

llms agents benchmarks

arXiv AI

Jul 9

Riemannian Geometry for Pre-trained Language Model Embeddings

arXiv:2607. 07047v1 Announce Type: cross Abstract: Understanding the geometric structure of pre-trained language model embeddings matters for interpretability and safety.

By Szczepan Konior, Alexandre Quemy, Przemys{\l}aw Klocek, Gr\'egoire Cattan, Bart{\l}omiej Sobieski

llms rag benchmarks safety

arXiv AI

Jul 9

Health System Scale Semantic Search Across Unstructured Clinical Notes

arXiv:2604. 25605v2 Announce Type: replace-cross Abstract: Introduction: Semantic search, which retrieves documents based on conceptual similarity rather than keywords, offers advantages for retrieval of clinical information.

By Faith Wavinya Mutinda, Spandana Makeneni, Anna Lin, Shivaji Dutta, Irit R. Rasooly, Patrick Dibussolo, Shivani Kamath Belman, Hessam Shahriari, Kevin Murphy, Alex B. Ruan, Barbara H. Chaiyachati, Sanjay Chainani, Robert W. Grundmeier, Scott M. Haag, Jeffrey M. Miller, Heather M. Griffis, Ian M. Campbell

llms rag benchmarks

arXiv AI

Jul 9

When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs

arXiv:2607. 07395v1 Announce Type: cross Abstract: Reliable confidence estimation remains a key limitation of test-time adaptation in vision-language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence.

By Tanay Sodha, Aditya Sharma, Ramya Hebbalaguppe, Vinti Agarwal, Pranav Murthy Yeluripaty

llms rag multimodal benchmarks

arXiv AI

Jul 9

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

arXiv:2607. 06799v1 Announce Type: cross Abstract: Evaluating uncertainty in AI-generated SQL queries requires estimating whether a query is correct, where correct means it executes to the same result as a human-written reference.

By Robert Richardson

llms fine-tuning benchmarks data-engineering

arXiv AI

Jul 9

Enhancing deep learning models for time series classification via knowledge distillation

arXiv:2607. 06796v1 Announce Type: cross Abstract: Deep learning has achieved remarkable success in various domains including time series analysis, computer vision and natural language processing.

By Javidan Abdullayev, Maxime Devanne, Jonathan Weber, Germain Forestier

llms computer-vision nlp efficiency benchmarks

arXiv Machine Learning

Jul 9

HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing

arXiv:2607. 06619v1 Announce Type: cross Abstract: Modern processor verification struggles to reach deep architectural states due to the inefficiencies of traditional mutation-based fuzzing.

By Ya Wang, Hanwei Fan, Zhenguo Liu, Xiaofeng Zhou, Yangdi Lyu, Jiang Xu, Wei Zhang

agents reinforcement-learning efficiency benchmarks

arXiv AI

Jul 9

Specification Grounding Drives Test Effectiveness for LLM Code

arXiv:2607. 06636v1 Announce Type: cross Abstract: Large language models frequently generate code that appears correct on typical inputs yet fails on edge cases, invalid inputs, and other specification-defined corner conditions.

By Amin Haeri, Mahdi Ghelichi

llms benchmarks

arXiv AI

Jul 9

Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks

arXiv:2607. 07189v1 Announce Type: new Abstract: Vision-language models (VLMs) and agentic AI have shown strong performance on semantic visual tasks, but it remains unclear whether they can handle the physics and inverse problems that underlie computational imaging.

By Ethan Chung, Chuanjun Zheng, Jasper Tan, Jingxi Li, Haopeng Zhang, Huaijin Chen

llms agents multimodal benchmarks

arXiv AI

Jul 9

AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning

arXiv:2607. 07033v1 Announce Type: cross Abstract: Large vision-language models incur substantial inference costs because high-resolution inputs introduce thousands of visual tokens, many of which are redundant for a given query.

By Kyuan Oh, Bumsoo Kim

llms efficiency multimodal benchmarks

arXiv AI

Jul 9

Navigating Hierarchy: Hyperbolic Learning on Brain Graphs for Disorder Diagnosis

arXiv:2607. 07077v1 Announce Type: cross Abstract: Functional brain networks exhibit a hierarchical organization across ROI, community, and whole-brain levels, supporting local processing, inter-community coordination, and global integration.

By Yapeng Li, Bo Jiang, Ziyan Zhang, Dongdong Chen, Zhengzheng Tu

benchmarks

arXiv Machine Learning

Jul 9

Nectar: Neural Estimation of Cached-Token Attention via Regression

arXiv:2605. 09778v2 Announce Type: replace Abstract: Evaluating softmax attention over a fixed long context requires reading every cached key-value pair for each new query token.

By Jo\~ao Monteiro, Michal Klein, Pierre Ablin, Marco Cuturi

benchmarks

arXiv Machine Learning

Jul 9

FPTQuant: Function-Preserving Transforms for LLM Quantization

arXiv:2506. 04985v2 Announce Type: replace Abstract: Large language models (LLMs) require substantial compute, and thus energy, at inference time.

By Boris van Breugel, Yelysei Bondarenko, Paul Whatmough, Markus Nagel

llms efficiency benchmarks

arXiv Machine Learning

Jul 9

Efficient Bayesian Deep Ensembles via Analytic Predictive Inference

arXiv:2607. 06776v1 Announce Type: new Abstract: We introduce an efficient Bayesian deep ensemble method for predictive regression designed to enhance interpretability while maintaining competitive predictive performance and computational efficiency.

By Sina Aghaee Dabaghan Fard, Marie Maros, Jaesung Lee

benchmarks safety

arXiv Machine Learning

Jul 9

MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models

arXiv:2607. 07673v1 Announce Type: cross Abstract: Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams.

By Hyunjae Kim, Dain Kim, Pan Xiao, Serina S. Applebaum, Younjoon Chung, Xuguang Ai, Yu Yin, Roy Jiang, Yuexi Du, Yawen Wei, Yiming Kong, Tuo Guo, Zhiyuan Cao, Mengmeng Du, Yuelei Fu, Yan Hu, Rui Shi, Gui Yang, Kevin W. Jin, Yuntian Liu, Yuxuan Tian, Jonathan Marquez, Zhen Chen, Sheng Zhang, Hoifung Poon, Hua Xu, Jaewoo Kang, Qingyu Chen

llms multimodal benchmarks safety

arXiv AI

Jul 9

EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models

arXiv:2602. 23802v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) have shown remarkable progress in visual reasoning and understanding tasks but still struggle to capture the complexity and subjectivity of human emotions.

By Yiyang Fang, Wenke Huang, Pei Fu, Yihao Yang, Kehua Su, Zhenbo Luo, Jian Luan, Mang Ye

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jul 9

AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

arXiv:2607. 06624v1 Announce Type: new Abstract: We present AgentLens, a production-assessed benchmark for interactive code agents.

By Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, Sergey Nikolenko

llms agents benchmarks

arXiv AI

Jul 9

Overview of the NLPCC 2026 Shared Task 1: Difficulty-Aware Multilingual and Multimodal Medical Instructional Video Understanding Evaluation

arXiv:2607. 06618v1 Announce Type: cross Abstract: Following the CMIVQA, MMI-VQA, and M4IVQA challenges in NLPCC 2023--2025, we introduce the Difficulty-Aware Medical Instructional Video Question Answering (DA-MIVQA) shared task for NLPCC 2026.

By Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Bin Li

nlp multimodal benchmarks

arXiv Machine Learning

Jul 9

PB-OEL: A Performance-Bounded Online Ensemble Learning Framework With Mixed Feedback for Real-Time Safety Assessment

arXiv:2503. 15581v2 Announce Type: replace Abstract: Real-time safety assessment is critical for ensuring the reliable operation of complex dynamic systems.

By Songqiao Hu, Zeyi Liu, Lufeng Hao, Yinzhong Cheng, Xiao He

benchmarks safety

arXiv AI

Jul 9

Operational Reframing and Approval-Framed Delegation in Multi-Agent LLM Safety

arXiv:2607. 07097v1 Announce Type: new Abstract: Safety evaluations of multi-agent LLM systems often compare a direct prompt with a planner-executor pipeline and report the difference as a single "pipeline effect.

By Lifei Liu, Haoran Yu, Xiaochong Jiang, Su Wang, Pin Qian, Yihang Chen

llms agents benchmarks safety

Get the daily briefing