Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,633 stories · RSS feed

arXiv AI

Jul 20

Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning

arXiv:2607. 16057v1 Announce Type: cross Abstract: Large language models (LLMs) are improving rapidly as reflected in benchmark scores, yet these AI benchmarks largely test capabilities such as factual recall, narrow question answering, mathematical problem-solving, and coding and agentic tool-use.

By Ajay Patel, Kartik Hosanagar, Ramayya Krishnan, Chris Callison-Burch, Karim Lakhani, Mitch Weiss

llms agents nlp benchmarks safety

arXiv AI

Jul 20

CAMMAR: Culture-Aware Matryoshka for Metaphorical Arabic Representations

arXiv:2607. 15847v1 Announce Type: cross Abstract: Metaphor in Arabic is a culturally grounded mechanism for constructing meaning, encoding cultural knowledge that shapes interpretation.

By Suzan Awinat, Alfonso Ortega del Puente

llms rag benchmarks

arXiv Machine Learning

Jul 20

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

arXiv:2607. 15942v1 Announce Type: cross Abstract: Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks.

By Stefan Maria Ailuro (INSAIT, Sofia University "St. Kliment Ohridski"), Mario Markov (INSAIT, Sofia University "St. Kliment Ohridski"), Mohammad Mahdi (INSAIT, Sofia University "St. Kliment Ohridski"), Luc Van Gool (INSAIT, Sofia University "St. Kliment Ohridski"), Danda Pani Paudel (INSAIT, Sofia University "St. Kliment Ohridski")

llms computer-vision reinforcement-learning multimodal benchmarks safety

arXiv AI

Jul 20

DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings

arXiv:2607. 15418v1 Announce Type: new Abstract: We introduce DrawingVQA, the first benchmark designed to evaluate multimodal large language models (MLLMs) on real-world construction drawings -- a core media in architecture, civil, and many other engineering practices.

By Yoonhwa Jung, Junryu Fu, Mani Golparvar-Fard

llms multimodal benchmarks

arXiv AI

Jul 20

IMBench: A Benchmark for Intuitive Robotic Manipulation

arXiv:2607. 15641v1 Announce Type: cross Abstract: Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints.

By Anurag Maurya, Sukhvansh Jain, Prajwal Avhad, Gautham Balachandran, Ziyi Zhou, Atharva Kshirsagar, Satyam Singh, Bowen Li. Rishabh Mukund, Ritul Singh, Jatin Vira, Suvonil Chatterjee, Devesh K. Jha

llms agents robotics multimodal benchmarks

arXiv Machine Learning

Jul 20

Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentation

arXiv:2502. 06818v4 Announce Type: replace Abstract: Recent works modify CLIP to perform open-vocabulary semantic segmentation in a training-free manner (TF-OVSS).

By Jingyun Wang, Cilin Yan, Guoliang Kang

rag computer-vision benchmarks

arXiv AI

Jul 20

CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data

arXiv:2607. 16122v1 Announce Type: new Abstract: Evaluations should do more than measure a models current performance.

By Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru, MohammadHossein Rezaei, Aakash Sabharwal, Yunzhong He

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 20

SC-JEPA: Stabilizing Latent Predictive Learning for Time-Series Anomaly Prediction

arXiv:2602. 04643v2 Announce Type: replace Abstract: Time-series anomaly prediction aims to forecast future system failures before they fully emerge, making latent predictive models such as JEPA a promising framework for capturing precursor dynamics.

By Yanan He, Yunshi Wen, Xin Wang, Tengfei Ma

diffusion efficiency benchmarks

arXiv AI

Jul 20

DECODEM: Data Extraction from Corporate Organizational Documents via Enhanced Methods

arXiv:2607. 15879v1 Announce Type: cross Abstract: Much empirical legal research depends on translating unstructured text into structured variables.

By Jens Frankenreiter

llms benchmarks

arXiv AI

Jul 20

Field-Aware RankMixer with Dual-Stream Bilinear Fusion for the Tencent UNI-REC Challenge

arXiv:2607. 15590v1 Announce Type: cross Abstract: This paper presents our solution to the KDD Cup 2026 Tencent UNIREC Challenge.

By Yufeng Zhang, Zhengqi Xu, Jiajun Cui

benchmarks

arXiv AI

Jul 20

Kolmogorov--Arnold Networks for Small Language Models

arXiv:2607. 15525v1 Announce Type: cross Abstract: Kolmogorov--Arnold Networks (KANs) replace fixed node activations with learned one-dimensional edge functions, offering an explicit interface for interpretation and a possible alternative to transformer feed-forward networks.

By Felippe Alves, Renato Vicente

llms efficiency benchmarks

arXiv AI

Jul 20

AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets

arXiv:2607. 15781v1 Announce Type: new Abstract: Geospatial datasets support applications from urban planning to climate modeling, yet consistent assessment of FAIR compliance is difficult.

By Ming Chen, Pranav Pai

llms agents benchmarks safety

arXiv AI

Jul 20

Cura 1T: Specialized Model for Agentic Healthcare

arXiv:2607. 15314v1 Announce Type: new Abstract: Healthcare spans high-stakes communication, expert reasoning, and workflow execution, yet specialized LLMs that cover these use cases together remain limited.

By actAVA AI, :, Haolin Chen, Leon Qi, Steve Brown, Deon Metelski, Tao Xia, Joonyul Lee, Qixuan Wang, Kevin Riley, Frank Wang, Weiran Yao

llms agents benchmarks

arXiv AI

Jul 20

Coercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalation

arXiv:2607. 15434v1 Announce Type: cross Abstract: Multi-agent systems routinely place one AI agent in authority over another.

By Jasmine Brazilek, Maheep Chaudhary, Zoe Lu, Miles Tidmarsh

llms agents benchmarks

arXiv AI

Jul 20

When Model Merging Rivals Joint Multi-Task Reinforcement Learning: A Task-Vector Geometry Analysis

arXiv:2607. 16062v1 Announce Type: cross Abstract: Model merging is promoted as a substitute for joint multi-task training, yet in the reinforcement-learning setting this substitution is essentially never tested against the baseline it claims to replace: methods merge independently released agents precisely because a joint model is unavailable.

By S. Aaron McClendon

agents reinforcement-learning benchmarks

arXiv AI

Jul 20

Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models

arXiv:2607. 15565v1 Announce Type: cross Abstract: Where should the question go in a vision-language model (VLM) prompt: before the image or after it?

By Rakshanda Hassan Abhinandan, John Galeotti, Deva Ramanan, Gautam Rajendrakumar Gare

llms nlp fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 20

Improving Backward Conformal Prediction via Non-Conformity Score Transformation

arXiv:2602. 01733v3 Announce Type: replace-cross Abstract: Conformal Prediction (CP) provides a statistical framework for uncertainty quantification that constructs prediction sets with coverage guarantees.

By Junxian Liu, Hao Zeng, Hongxin Wei

benchmarks

arXiv Machine Learning

Jul 20

ASK-NN: An Asymmetric Nearest-Neighbor Test that detects Distribution Drifts in Natural Language

arXiv:2607. 15607v1 Announce Type: new Abstract: Hallucinations and artificial text in LLM-generated outputs often appear as distributional deviations between prompt and response hidden-state distributions.

By Sergey Zakharov, Rodion Oblovatny, Alexey Zaytsev

llms benchmarks safety

arXiv AI

Jul 20

LVSum: A Benchmark for Timestamp-Aware Long Video Summarization

arXiv:2604. 10024v2 Announce Type: replace-cross Abstract: Long video summarization presents significant challenges for multimodal large language models (MLLMs), particularly in maintaining temporal fidelity over extended durations and producing summaries that are both semantically and temporally grounded.

By Alkesh Patel, Melis Ozyildirim, Ying-Chang Cheng, Ganesh Nagarajan

llms nlp multimodal benchmarks safety

arXiv Machine Learning

Jul 20

LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models

arXiv:2607. 15447v1 Announce Type: new Abstract: Recent research in clinical machine learning, focusing on outcome predictions in intensive care unit (ICU), has shifted from bespoke supervised models to foundation models, utilising modern representation learning methods.

By Jingteng Li, Alexander Capstick, Louise Rigny, Iona Biggart, Neil J Sebire, Payam Barnaghi

llms rag benchmarks

Get the daily briefing