Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,077 stories · RSS feed

arXiv AI

Jul 21

CLOSER-Bench: Evaluating Budgeted Cross-Stage Design Closure for Hardware Agents

arXiv:2607. 16632v1 Announce Type: cross Abstract: Hardware engineering exposes coding agents to a form of long-horizon work that is difficult to capture with pass-at-k: progress is continuous, tool feedback is delayed and heterogeneous, and a backend failure may require revising RTL rather than tuning another physical-design parameter.

By Peilong Zhou, Zhirong Chen, Cangyuan Li, Haoyu Gao, Kaiyan Chang, Ziming Qu, Ying Wang

agents benchmarks

arXiv Machine Learning

Jul 21

Singular Bayesian Neural Networks

arXiv:2602. 00387v4 Announce Type: replace-cross Abstract: Bayesian neural networks promise calibrated uncertainty but require $O(mn)$ parameters for standard mean-field Gaussian posteriors.

By Mame Diarra Toure, David A. Stephens

llms benchmarks safety

arXiv AI

Jul 21

CDIS: Cross-Dimensional Class-Agnostic 3D Instance Segmentation via 2D Mask Tracking and 3D-2D Projection Merging

arXiv:2607. 17778v1 Announce Type: cross Abstract: Class-agnostic 3D instance segmentation is critical for robotic systems operating in unknown environments, enabling perception of previously unseen objects for reliable manipulation and navigation.

By Juno Kim, Hye-Jung Yoon, Yesol Park, Byoung-Tak Zhang

computer-vision robotics benchmarks

arXiv AI

Jul 21

Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generation

arXiv:2607. 18029v1 Announce Type: cross Abstract: Researchers need to answer ad-hoc questions about the contents of domain-specific archives but often lack the expertise to write structured queries on the metadata.

By Blake G. Fitch, Cato Elia Kurtz

llms agents fine-tuning benchmarks safety data-engineering

arXiv AI

Jul 21

A Survey on Unlearnable Data

arXiv:2503. 23536v3 Announce Type: replace-cross Abstract: Unlearnable data (ULD) has emerged as an innovative defense technique to prevent machine learning models from learning meaningful patterns from specific data, thus protecting data privacy and security.

By Jiahao Li, Yiqiang Chen, Yunbing Xing, Yang Gu, Xiangyuan Lan

benchmarks safety

arXiv AI

Jul 21

CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

arXiv:2603. 27958v2 Announce Type: replace Abstract: Analogical reasoning tests a fundamental aspect of human cognition: mapping the relation from one pair of objects to another.

By Yongkang Du, Xiaohan Zou, Minhao Cheng, Lu Lin

llms multimodal benchmarks

arXiv AI

Jul 21

SONAR: Spectral-Contrastive Audio Residuals for Generalizable Deepfake Detection

arXiv:2511. 21325v2 Announce Type: replace-cross Abstract: Deepfake (DF) audio detectors still struggle to generalize to out of distribution inputs.

By Ido Nitzan Hidekel, Gal lifshitz, Khen Cohen, Dan Raviv

rag diffusion benchmarks safety

arXiv AI

Jul 21

Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage

arXiv:2601. 03043v4 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference efficiency.

By Junhao Hu, Fangze Li, Mingtao Xu, Feifan Meng, Shiju Zhao, Tiancheng Hu, Ting Peng, Anmin Liu, Wenrui Huang, Chenxu Liu, Ziyue Hua, Tao Xie

llms benchmarks

arXiv AI

Jul 21

BBOPlace-Bench: Benchmarking Black-Box Optimization for Chip Placement

arXiv:2510. 23472v2 Announce Type: replace-cross Abstract: Chip placement is a vital stage in modern chip design, and black-box optimization (BBO) has been applied to it for decades.

By Ke Xue, Ruo-Tong Chen, Rong-Xi Tan, Xi Lin, Yunqi Shi, Siyuan Xu, Mingxuan Yuan, Chao Qian

reinforcement-learning benchmarks

arXiv AI

Jul 21

Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing

arXiv:2509. 14335v2 Announce Type: replace-cross Abstract: Automated malware classifiers achieve strong detection performance, but auditing requires more than flagging a sample: analysts must explain malicious behaviors and justify them with code evidence.

By Xinran Zheng, Xingzhi Qian, Yiling He, Shuo Yang, Lorenzo Cavallaro

llms agents benchmarks

arXiv AI

Jul 21

Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering

arXiv:2510. 01483v3 Announce Type: replace-cross Abstract: Vision-language models (VLMs) demonstrate strong image-level scene understanding, but reasoning over long egocentric video remains costly: because VLMs maintain no persistent memory or explicit spatial representation, all sampled frames must be re-processed for every new query.

By Mohamad Al Mdfaa, Svetlana Lukina, Timur Akhtyamov, Arthur Nigmatzyanov, Dmitrii Nalberskii, Sergey Zagoruyko, Gonzalo Ferrer

llms agents nlp robotics multimodal benchmarks

arXiv AI

Jul 21

Short-Term Electricity Demand Forecasting for New England: A Comprehensive Machine Learning Benchmark with Weather, Calendar, and COVID-19 Indicators

arXiv:2606. 20918v2 Announce Type: replace-cross Abstract: Accurate short-term electricity demand forecasting is critical for reliable power system operation, energy market planning, and infrastructure optimization.

By Reza Ghanavati, Behrooz Mosallaei

llms benchmarks

arXiv AI

Jul 21

A Benchmark for Early-stage Parkinson's Disease Detection from Speech

arXiv:2605. 14066v2 Announce Type: replace-cross Abstract: Early-stage Parkinson's disease (EarlyPD) detection from speech is clinically meaningful yet underexplored, and published results are hard to compare because studies differ in datasets, languages, tasks, evaluation protocols, and EarlyPD definitions.

By Terry Yi Zhong, Cristian Tejedor-Garcia, Khiet P. Truong, Janna Maas, Louis ten Bosch, Bastiaan R. Bloem

benchmarks

arXiv AI

Jul 21

When Cultures Move: Measuring and Improving Multicultural Text-to-Video Generation

arXiv:2605. 16716v5 Announce Type: replace-cross Abstract: Text-to-video (T2V) generation has rapidly progressed in visual fidelity, yet its ability to faithfully represent multiple cultures within a single prompt remains underexplored.

By Shuowei Li, Yuming Zhao, Parth Bhalerao, Oana Ignat

agents diffusion multimodal benchmarks

arXiv AI

Jul 21

Do Maps Still Matter for Machines: Revisiting the Role of Choropleth Maps in Foundation Model Spatial Understanding

arXiv:2607. 17999v1 Announce Type: new Abstract: Spatial understanding is crucial for foundation models (FMs), and maps have long helped humans organize and reason about geographic information.

By Zhiwei Wei, Yonghe Sun, Zhenjia Liu, Wenjia Xu, Chao He, Weihua Dong, Chunbo Liu, Hua Liao

llms benchmarks

arXiv AI

Jul 21

Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget

arXiv:2607. 13125v2 Announce Type: replace-cross Abstract: We introduce Boogu-Image-0.

By Guoxuan Chen, Chufeng Xiao, Haoran Yang, Siyue Xie, Binxiao Huang, Ming Zhang, Cheuk Him Chau, Xinyu Fu, Yingzhao Lian, Tom S. Y. Li, Jintao Lin, Bowen Dong, Zian Qian, Yuhao Liu, Yuxuan Hu, Weikang Shi, Bin Zou, Bowen Zheng, Haoxuan Che, Chang Chen, Yuyang He, Heyang Sun, Tianyu Huang, Chong Hou Choi, Cheng Gong, Han Shi, Haoli Bai, Xihui Liu, Hongsheng Li, Qifeng Chen, Chao Huang, Rui Liu, Chenyang Lei

llms agents diffusion multimodal benchmarks data-engineering

arXiv Machine Learning

Jul 21

AOE: Exhaustive Out-of-Distribution Detection via Recalibrating Outlier Labels

arXiv:2605. 28021v2 Announce Type: replace Abstract: Out-of-distribution (OOD) detection is essential for deploying machine learning models in open-world and safety-critical scenarios, where test inputs may deviate from the training distribution and overconfident predictions on unknown samples can lead to unreliable decisions.

By Fengqiang Wan, Qing-Yuan Jiang, Fu Shen, Yang Yang

benchmarks safety

arXiv AI

Jul 21

JOR-Bench: Japanese Operations Research Benchmarks for Large Language Models

arXiv:2607. 16777v1 Announce Type: cross Abstract: We present JOR-Bench, a collection of five Japanese-language benchmarks for evaluating the ability of large language models (LLMs) to formulate and solve operations research (OR) problems.

By Yuu Jinnai

llms benchmarks

arXiv AI

Jul 21

What Do They See? Interpreting Complex Road Scenarios Through the Eyes of Vision-Language-Action Models for Safe and Trustworthy Autonomous Vehicle Learning

arXiv:2607. 16938v1 Announce Type: cross Abstract: End-to-end autonomous driving models are now able to navigate complex road scenarios, mapping raw sensor observations directly to observed paths for open-loop evaluation and often effective driving in closed-loop evaluation.

By Kalpana Panda, Wesley Maia, Vinti Agarwal, Ross Greer

agents robotics multimodal benchmarks safety

arXiv AI

Jul 21

Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent

arXiv:2607. 17044v1 Announce Type: cross Abstract: Multi-step enterprise agent tasks fail in a characteristic way: single-pass inference has no checkpoint between deciding an answer and committing to it.

By Arunabh Dastidar (for the Leni Team)

agents benchmarks

Get the daily briefing