Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,859 stories · RSS feed

arXiv AI

Jun 15

SANA: What Matters for QA Agents over Massive Data Lakes?

arXiv:2606. 13904v1 Announce Type: cross Abstract: Exploratory question answering (EQA) over data lakes requires an LLM agent to discover relevant sources, analyze retrieved data, and adapt its actions based on intermediate results.

By Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang, Eugene Wu

llms agents nlp benchmarks

arXiv Machine Learning

Jun 15

Direct Fisher Score Estimation for Likelihood Maximization

arXiv:2506. 06542v2 Announce Type: replace-cross Abstract: We study the problem of likelihood maximization when the likelihood function is intractable but model simulations are readily available.

By Sherman Khoo, Yakun Wang, Song Liu, Mark Beaumont

benchmarks safety

arXiv AI

Jun 15

UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

arXiv:2504. 20734v5 Announce Type: replace-cross Abstract: Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries.

By Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Baek, Sung Ju Hwang

rag fine-tuning benchmarks

arXiv AI

Jun 15

From Shield to Target: Denial-of-Service Attacks on LLM-Based Agent Guardrails

arXiv:2606. 14517v1 Announce Type: cross Abstract: LLM-based guardrails have emerged as a highly effective defense against prompt injection and jailbreak attacks in autonomous agents.

By Yuguang Zhou, Xunguang Wang, Pingchuan Ma, Zhantong Xue, Zhaoyu Wang, Shuai Wang

llms agents benchmarks safety

arXiv AI

Jun 15

EmoMind: Decoding Affective Captions from Human Brain fMRI

arXiv:2605. 16739v2 Announce Type: replace-cross Abstract: Decoding visual experience from brain activity has advanced substantially, but current brain-to-text systems largely recover semantic content while discarding affect.

By Bilal A. Mohammed, Lin Gu, Ruogu Fang

llms benchmarks

arXiv AI

Jun 15

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

arXiv:2606. 14249v1 Announce Type: new Abstract: AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts.

By Tingyang Chen, Shuo Lu, Kang Zhao, Weicheng Meng, Hanlin Teng, Tianhao Li, Chao Li, Xule Liu, Jian Liang, Zhizhong Zhang, Yuan Xie, Heng Qu, Kun Shao, Jian Luan

agents reinforcement-learning benchmarks

arXiv AI

Jun 15

SAFformer:Improving Spiking Transformer via Active Predictive Filtering

arXiv:2605. 08270v2 Announce Type: replace-cross Abstract: Spiking Neural Networks (SNNs) offer notable advantages in biological plausibility and energy efficiency, making them promising candidates for building low-power Transformers.

By Zequan Xie, Weiming Zeng, Yunhua Chen, Sichang Ling, Tongyang Chen, Jinsheng Xiao

llms benchmarks

arXiv AI

Jun 15

A Deep Reinforcement Learning (DRL)-Based Transformer Method for Solving the Open Shop Scheduling Problem

arXiv:2606. 13682v1 Announce Type: new Abstract: The open shop scheduling problem (OSSP) arises in many industrial and service settings but remains computationally challenging as the number of jobs and machines increases.

By Faezeh Ardali, Mwembezi A. Nyelele, Gerald M. Knapp

llms reinforcement-learning benchmarks

arXiv AI

Jun 15

Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources

arXiv:2606. 14141v1 Announce Type: cross Abstract: Sound events are entities with semantic identities, locations, and trajectories, but current audio-language models usually reason about clips as global event content.

By Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida, Kim Sung-Bin, Toshimitsu Uesaka, Takashi Shibuya, Kyeongyoon Lee, Tae-Hyun Oh, Yuki Mitsufuji

llms benchmarks

arXiv AI

Jun 15

A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets

arXiv:2606. 13802v1 Announce Type: cross Abstract: Predictive code completion greatly accelerates how quickly developers work.

By Tejas Agrawal, Vu Le, Sumit Gulwani, Gust Verbruggen

llms fine-tuning benchmarks

arXiv AI

Jun 15

Selective Agentic Recovery for UAV Autonomy with a Persistent Mission Runtime

arXiv:2606. 14219v1 Announce Type: cross Abstract: Agentic AI can support unmanned aerial vehicle (UAV) autonomy by providing high-level recovery reasoning when local waypoint- or setpoint-based execution encounters blocked passages, repeated no-progress behavior, or mission-level ambiguity.

By Taewoo Park, Kyeonghyun Yoo, Seunghyun Yoo, Hwangnam Kim

agents benchmarks safety

arXiv AI

Jun 15

TwinBI: An Agentic Digital Twin for Efficient Augmented Interactions with Business Intelligence Dashboards

arXiv:2606. 13731v1 Announce Type: new Abstract: Business intelligence (BI) increasingly combines dashboard interaction with LLM-based assistance, but these two modes often fall out of sync during multi-step analysis.

By Jisoo Jang Wen-Syan Li

llms agents robotics benchmarks data-engineering

arXiv AI

Jun 15

An integrated interpretable control effectiveness learning and nonlinear control allocation methodology for overactuated aircrafts

arXiv:2606. 13794v1 Announce Type: cross Abstract: Nonlinear dynamics and the strong couplings that arise between multiple effectors undermine the assumptions behind conventional, linear control allocation techniques.

By Umut Demir, Aamir Ahmad, Walter Fichter

benchmarks safety

arXiv AI

Jun 15

COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers

arXiv:2512. 02318v4 Announce Type: replace-cross Abstract: This paper studies how multimodal large language models (MLLMs) undermine the security guarantees of visual CAPTCHA.

By Junyu Wang, Changjia Zhu, Yuanbo Zhou, Lingyao Li, Xu He, Mingkui Wei, Junjie Xiong

llms multimodal benchmarks

arXiv AI

Jun 15

Unsupervised Learning of Efficient Exploration: Pre-training Adaptive Policies via Self-Imposed Goals

arXiv:2601. 19810v2 Announce Type: replace-cross Abstract: Unsupervised pre-training can equip reinforcement learning agents with prior knowledge and accelerate learning in downstream tasks.

By Octavio Pappalardo

agents reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 15

MoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition

arXiv:2606. 14459v1 Announce Type: cross Abstract: Modern Automatic Speech Recognition (ASR) systems have made remarkable progress on standard benchmarks, yet performance gaps have emerged under real-world distribution shifts, caused by recording conditions, accents, speech impairments, and noise.

By Theresa Pekarek Rosin, Matthias Kerzel, Stefan Wermter

multimodal benchmarks

arXiv AI

Jun 15

Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

arXiv:2606. 14516v1 Announce Type: new Abstract: AI evaluations are widely used for testing and understanding progress.

By Jan Batzner, Sree Harsha Nelaturu, Anastassia Kornilova, Jon Crall, Tommaso Cerruti, Yanan Long, Yifan Mai, Sanchit Ahuja, Asaf Yehudai, Marek \v{S}uppa, John P. Lalor, Oluwagbemike Olowe, Jatin Ganhotra, Brian H. Hu, Eliya Habba, Andrew M. Bean, Chang Liu, Sander Land, Steven Dillmann, Aniketh Garikaparthi, Elron Bandel, Saki Imai, James Edgell, Wm. Matthew Kennedy, Jenny Chim, Patrick Meusling, Asteria Kaeberlein, Venkata Ramachandra Karthik Chundi, Manasi Patwardhan, Martin Ku, Austin Meek, Leon Knauer, Brian Wingenroth, Srishti Yadav, Usman Gohar, Felix Friedrich, Michelle Lin, Jennifer Mickel, Arman Cohan, Stella Biderman, Irene Solaiman, Zeerak Talat, Anka Reuel, Mubashara Akhtar, Gjergji Kasneci, Avijit Ghosh, Leshem Choshen

benchmarks

arXiv Machine Learning

Jun 15

XRDiff: Crystal Structure Prediction from Powder X-Ray Diffraction Data Using Diffusion Models

arXiv:2606. 14003v1 Announce Type: cross Abstract: Determining the crystal structure of a material from its powder X-ray diffraction (PXRD) pattern is a central challenge in materials science.

By Nofit Segal, Mingda Li, Benjamin Kurt Miller, Rafael G\'omez-Bombarelli

diffusion benchmarks

arXiv AI

Jun 15

A Comparative Study of Deep Learning Architectures for Multi-Horizon Behavioural Forecasting for Mobile Health

arXiv:2606. 14604v1 Announce Type: cross Abstract: Wearable devices and smartphones generate rich behavioural time series that can support proactive health interventions, yet systematic comparisons of modern forecasting architectures for these data are lacking.

By Pavlos Nicolaou, Kleanthis Malialis, Artemis Kontou, Panayiotis Kolios

llms fine-tuning benchmarks

arXiv AI

Jun 15

Explaining RhythmFormer: A Systematic XAI Analysis of Periodic Sparse Attention for Remote Photoplethysmography

arXiv:2606. 13839v1 Announce Type: cross Abstract: Remote photoplethysmography (rPPG) transformers achieve low heart-rate error on benchmarks, yet their decisions remain opaque--a growing concern as rPPG moves toward clinical heart rate estimation.

By Louis Chen, Torbj\"orn E. M. Nordling

llms benchmarks safety

Get the daily briefing