Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,030 stories · RSS feed

arXiv AI

Aug 11

Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention

arXiv:2608. 07921v1 Announce Type: cross Abstract: We apply Marchenko-Pastur (MP) random matrix theory to pre-trained attention weights in order to separate each projection matrix into a random-like bulk and a set of spectral outliers.

By Kasun Dewage, Marianna Pensky, Suranadi De Silva, T. H. Bandara

llms fine-tuning efficiency benchmarks

arXiv AI

Aug 11

A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization

arXiv:2608. 08180v1 Announce Type: cross Abstract: Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events.

By Praveen Kumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala, Naman Kabadi

agents nlp benchmarks safety

arXiv AI

Aug 11

Quokka: Accelerating Program Verification with LLMs via Invariant Synthesis

arXiv:2509. 21629v4 Announce Type: replace-cross Abstract: Program verification relies on loop invariants, yet automatically discovering strong invariants remains a long-standing challenge.

By Anjiang Wei, Tianran Sun, Tarun Suresh, Haoze Wu, Ke Wang, Alex Aiken

llms fine-tuning benchmarks

arXiv AI

Aug 11

CADEngBench: It Looks Like CAD, but Does It Work? Evaluating Parametric Design, Assembly Reasoning, and Physics Simulation

arXiv:2608. 09296v1 Announce Type: new Abstract: A CAD model is not engineering-grade merely because it looks correct.

By Harmanjot Singh, Abhra Dubey, Jorge Alejandro Amador Herrera

multimodal benchmarks

arXiv AI

Aug 11

Towards Researcher Agents for Knowledge-Graph Question Answering

arXiv:2608. 07700v1 Announce Type: new Abstract: Translating a natural-language question into a SPARQL query that can be executed against a large knowledge graph requires resolving lexical ambiguity, grounding surface terms in the target ontology, and producing graph patterns that are both syntactically valid and semantically faithful.

By Tommaso Soru, Abdulsobur Oyewale

agents nlp benchmarks

arXiv AI

Aug 11

Directed Neuro-Symbolic Stochastic Execution for Verification of Distributed Parallel AI Programs

arXiv:2608. 07947v1 Announce Type: new Abstract: Distributed parallel Artificial Intelligence (AI) programs expose reliability gaps that conventional testing cannot close: parallel executions are non-deterministic, and AI workloads bring high-dimensional inputs and non-linear operations that defeat fuzzing and symbolic execution in isolation.

By Gautham Koorma, Vikas Sharma, George Edwards, Mahdi Eslamimehr

llms benchmarks

arXiv AI

Aug 11

BRACE: Taming Sharp Irregularities via Barycentric Rational Forecasting for Fast Diffusion Transformers Inference

arXiv:2608. 07572v1 Announce Type: cross Abstract: Diffusion Transformers (DiTs) have demonstrated exceptional performance in high-fidelity image and video generation.

By Jinlong Yang, Jinke Wu, Lizilin, Yao Zhou

llms diffusion benchmarks

arXiv AI

Aug 11

Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

arXiv:2608. 08160v1 Announce Type: cross Abstract: The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling.

By Yingpeng Ma, Jianhao Yan, Bei Shi, Ka Hou Kam, Runnan Wang, Xuebo Liu, Yulong Chen, Yue Zhang, Derek F. Wong

llms agents benchmarks safety

arXiv AI

Aug 11

SkillConsist: Detecting Inconsistencies in Agent Skills via Bidirectional Graph Alignment

arXiv:2608. 07639v1 Announce Type: cross Abstract: Agent Skills provide reusable capabilities to LLM agents.

By Chaofan Meng, Yuhang Zheng, Yingnan Zhou, Sihan Xu

llms agents benchmarks safety

arXiv AI

Aug 11

PROSLEX: A Novel Dataset for Expert-Annotated Legal Statute Prediction for Indian Judiciary

arXiv:2608. 08830v1 Announce Type: new Abstract: Legal Statute Prediction (LSP) involves automatically identifying relevant legal statutes given factual descriptions in legal documents, typically framed as a multi-label classification task within natural language processing and information retrieval research.

By Subinay Adhikary, Upal Bhattacharya, Vivek Kumar Singh, Anurag Sharma, Shubham Kumar Nigam, Suvasis Das, Shouvik Kumar Guha, Koustav Rudra, Kripabandhu Ghosh

llms nlp benchmarks

arXiv AI

Aug 11

Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents

arXiv:2608. 08852v1 Announce Type: new Abstract: AI agents can now solve problems, answer like subject experts, and generate long-form multimodal content.

By Yi-Cheng Lin, Yu-Kai Guo, Szu-Chi Chen, Bo-Han Feng, Yun-Man Hsu, Hsiang Hsieh, Yu-Jung Lin, Yue-Ling Wu, Jia-Kai Dong, An-Yu Cheng, Yu-Han Huang, Lok-Lam Ieong, Kuan-Yu Chen, Ming-Douo Tchouang, Shao-Hua Sun, Che Lin, Jian-Jiun Ding, Hung-yi Lee

llms agents diffusion multimodal benchmarks

arXiv AI

Aug 11

Beyond Pixels: Benchmarking and Reward-Based Assessing Framework for Visual Spatial Aesthetics

arXiv:2512. 05098v2 Announce Type: replace-cross Abstract: In recent years, Image Quality Assessment (IQA) for AI-generated images (AIGI) has advanced rapidly; however, existing methods primarily target portraits and artistic images, lacking a systematic evaluation of interior scenes.

By Yuan Gao, Jin Song, Yiyun Fei, Gongzhe Li, Ruigao Yang

reinforcement-learning fine-tuning benchmarks

arXiv AI

Aug 11

GALA: Graph-Augmented LLM Agents for Root Cause Analysis and Incident Response in Microservices

arXiv:2608. 08968v1 Announce Type: cross Abstract: Microservice root cause analysis (RCA) requires correlating failures across heterogeneous telemetry within complex service dependency graphs.

By Yifang Tian, Yaming Liu, Zichun Chong, Zihang Huang, Yiran Li, Hans-Arno Jacobsen

llms agents multimodal benchmarks safety

arXiv AI

Aug 11

Probabilistic Circuits for Knowledge Graph Completion with Reduced Rule Sets

arXiv:2508. 06706v2 Announce Type: replace Abstract: Rule-based methods for knowledge graph completion provide explainable results, but often require tens of thousands of rules to achieve competitive performance.

By Jaikrishna Manojkumar Patil, Nathaniel Lee, Al Mehdi Saadat Chowdhury, YooJung Choi, Paulo Shakarian

benchmarks

arXiv AI

Aug 11

Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains

arXiv:2608. 09873v1 Announce Type: cross Abstract: We introduce Sci-VBench, a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains.

By Diandian Zhang, Tingyu Song, Lin Fu, Zheyuan Yang, Yilun Zhao

diffusion benchmarks

arXiv AI

Aug 11

Coupled Graph--Policy Distillation for Personalized Medication Safety in Older Adults with Multimorbidity

arXiv:2608. 09443v1 Announce Type: new Abstract: Large language model (LLM) agents can support medication review between clinical visits, but safe choices for older adults with multimorbidity depend on conditions, medications, and geriatric risks that users may omit.

By Zihan Wang, Anglin Liu, Rongyi Wang, Dantong Li, Yi Lu, Siqing Yuan, Hongxia Xu, Zhongtian Long, Jintai Chen

llms agents efficiency benchmarks safety

arXiv Machine Learning

Aug 11

No Unique Minimizer, No Problem: On the Consistency of Robust Neural Classifiers

arXiv:2608. 08489v1 Announce Type: new Abstract: Neural network classifiers trained by cross-entropy minimization are highly sensitive to label noise and adversarial contamination.

By Subhabrata Majumdar, Anand Deo, Partha Pratim Saha, Abhik Ghosh

benchmarks safety

arXiv Machine Learning

Aug 11

Hierarchical rank-evolving representation for physics-informed neural networks

arXiv:2608. 09483v1 Announce Type: new Abstract: Recently, tensor-based physics-informed neural networks (T-PINNs) have received increasing attention.

By Ruoyang Su, Xi-Le Zhao, Kun Li, Liang Li

benchmarks

arXiv Machine Learning

Aug 11

Mechanistic Interpretability-Guided Selective Fine-Tuning of Vision-Language Models for Centimeter-Level Flood Depth Estimation

arXiv:2608. 07562v1 Announce Type: cross Abstract: Urban flooding poses an escalating threat to transportation infrastructure, yet no operational system provides real-time, street-level flood-depth estimates at centimeter resolution.

By Nafis Fuad, Xiaodong Qian, Dongxiao Zhu

llms computer-vision fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Aug 11

FedOrbit: Adaptive Personalized Federated Learning for Non-IID LEO Satellite Constellations

arXiv:2608. 09687v1 Announce Type: new Abstract: Federated learning (FL) in Low Earth Orbit (LEO) satellite constellations is affected by non-IID data and irregular ground-station visibility, both driven by orbital geometry.

By Satwat Bashir, Tasos Dagiuklas, Muddesar Iqbal

benchmarks

Get the daily briefing