Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

13,478 stories · RSS feed

arXiv AI

Aug 13

Gaze Target Estimation Anywhere with Concepts

arXiv:2608. 11367v1 Announce Type: cross Abstract: Estimating human gaze targets from images in-the-wild is an important and formidable task.

By Xu Cao, Houze Yang, Vipin Gunda, Zhongyi Zhou, Tianyu Xu, Adarsh Kowdle, Inki Kim, James M. Rehg

llms benchmarks

arXiv AI

Aug 13

TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs

arXiv:2608. 11415v1 Announce Type: cross Abstract: Large language models are being proposed as agents in scientific workflows, in domains where no downstream verifier exists.

By Valentin Rodionov, Shamil Assylbekov

llms agents benchmarks safety

arXiv AI

Aug 13

Do Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code Generation

arXiv:2608. 11513v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly integrated into software engineering workflows, helping developers write, debug, test, and maintain code.

By Alex Deaconu, Anubhav Gupta, Manaal Basha, Nicholas Haydu, Gema Rodr\'iguez-P\'erez

llms benchmarks

arXiv AI

Aug 13

Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentation

arXiv:2608. 11681v1 Announce Type: cross Abstract: This work addresses the challenge of open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS), which aim to recognize both predefined and unseen object categories without exhaustive human annotations.

By Duy Tran Thanh, Yeejin Lee, Byeongkeun Kang

llms computer-vision multimodal benchmarks safety

arXiv AI

Aug 13

G0.5: One Autoregressive Stream for Robot Reasoning and Action

arXiv:2608. 11739v1 Announce Type: cross Abstract: The prevailing recipe for Vision-Language-Action (VLA) models couples a pretrained VLM with a separately trained flow-matching action expert.

By Yicheng Liu, Zibin Dong, Baijun Ye, Tianyuan Yuan, Tao Jiang, Anqi Yang, Shicheng Cao, Haonan Liu, Yue Sun, Zihan Guo, Xiao Liu, Dong Ke, Changxun Pan, Chenru Wu, Tailai Cheng, Xiaoshu Ren, Xinlei Zhang, Jianning Cui, Zijie Zhao, Haoyu Zhang, Kaiming Xu, Haodong Yang, Bowen Zhang, Jiahui Niu, Shaoting Zhu, Shiduo Zhang, Hang Zhao

llms nlp robotics fine-tuning multimodal benchmarks

arXiv AI

Aug 13

JieZi: A Large-Scale Expert-Audited Dataset and Benchmark for Ancient Chinese Character Exegesis

arXiv:2608. 11741v1 Announce Type: cross Abstract: The scholarly exegesis of ancient Chinese characters demands integrating visual observation, linguistic analysis, and historical context.

By Ran Li, Huiguo He, Jiahuan Cao, Junle Liu, Hiuyi Cheng, Lianwen Jin

llms nlp fine-tuning multimodal benchmarks

arXiv AI

Aug 13

Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System

arXiv:2608. 11738v1 Announce Type: cross Abstract: Multimodal Large Language Model (MLLM)-based UAV aerial image understanding and reasoning is essential for aerial intelligence yet poses distinct challenges arising from extreme scale variation, arbitrary camera orientations, and high object density.

By Haoyu Zhang, Shuoxun Zhang, Peng Ye, Lin Zhang, Jiakang Yuan, Shenghong Yi, Yuening Wang, Tao Chen

llms agents multimodal benchmarks

arXiv AI

Aug 13

Benchmark-Based Comparative Assessment of Publicly Benchmarked Indian Foundation Models: A Capability and Evaluation-Maturity Framework

arXiv:2608. 11891v1 Announce Type: cross Abstract: Governments increasingly fund indigenous foundation models to strengthen national AI capability, digital sovereignty, and multilingual computing.

By Avinash Agarwal, Vridhi Jain

agents multimodal benchmarks

arXiv AI

Aug 13

CoQui: A Coordinate-Conditioned Quantum Implicit Generative Adversarial Network for End-to-End Image Generation

arXiv:2608. 11884v1 Announce Type: cross Abstract: Quantum generative adversarial networks (QGANs) have attracted increasing attention for image generation using parameterized quantum circuits.

By Xue Yang, Rigui Zhou, ShiZheng Jia, Dax Enshan Koh, Siong Thye Goh, Young-Wook Cho, YaoChong Li, Xuezhi Ma, Hongyu Chen, Xin Wang

rag diffusion benchmarks safety

arXiv AI

Aug 13

Two-Stage Deformable-Convolutional Inverse Design of Nanophotonic Absorbers from Optical Spectra

arXiv:2608. 11860v1 Announce Type: cross Abstract: Data-driven inverse design enables efficient generation of nanophotonic structures with prescribed optical responses, but spectrum-to-geometry mapping remains challenging due to non-uniqueness and fine geometric features.

By Waleed Waseer, Muhammad Shahid Jabbar, Muhammad Sohail Ibrahim, Shujaat Khan

benchmarks safety

arXiv AI

Aug 13

HCGRec: Hint-Conditioned Generative Recommendation with Semantic IDs

arXiv:2608. 11980v1 Announce Type: cross Abstract: Semantic-ID generative recommenders represent each item as a short sequence of discrete semantic tokens and predict the next item by autoregressively generating this token sequence.

By Kangning Zhang, Haotian Fang, Xukun Luo, Hao Yin, Yang Gao, Peng Yan, Weiwen Liu, Weinan Zhang, Yong Yu

fine-tuning benchmarks safety

arXiv AI

Aug 13

Faithful, Sufficient and Understandable: Rethinking Graph Counterfactual Explanations via Discrete Diffusion Inversion

arXiv:2608. 12083v1 Announce Type: cross Abstract: Graph Neural Networks (GNNs) achieve strong predictive performance on graph-structured data across domains such as chemistry, biology, and network analysis, yet they provide no intrinsic explanation of their predictions.

By David Bechtoldt, Sidney Bender

diffusion benchmarks safety

arXiv AI

Aug 13

M-Net: Integrating Spectral Features and Physical Field Operators into Deep Learning for Medical Image Segmentation

arXiv:2608. 12196v1 Announce Type: cross Abstract: Purpose: Deep learning-based medical image segmentation has achieved remarkable success, yet purely data-driven approaches often fail to exploit the rich mathematical structure inherent in medical images.

By Jing Zhu, Ye Wang, Fumin Wang

computer-vision benchmarks

arXiv AI

Aug 13

VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

arXiv:2608. 12246v1 Announce Type: cross Abstract: Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases.

By Jin Lu, Xuening Han, Yang Zhong, Lin Tan, Kevin Luo, Andrew Gacek, Neha Rungta

agents benchmarks

arXiv AI

Aug 13

Class Activation Mapping in Explainable Computer Vision: A Method-Centered Review of CNN, Transformer, and Foundation-Model-Era Visual Explanations

arXiv:2608. 12299v1 Announce Type: cross Abstract: Class activation mapping (CAM) is one of the most widely used visual explanation families in explainable artificial intelligence.

By AmirHossein Eshghi, Hamid Saadatfar, Seyyed Ali Hoseini, AmirMohsen Eshghi, Siavash Arjomand Bigdel

llms computer-vision benchmarks

arXiv AI

Aug 13

Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence

arXiv:2608. 12290v1 Announce Type: cross Abstract: Modern black-box Image-to-Video (I2V) models offer powerful capabilities in automated content creation, yet their lack of fine-grained control and reliability presents significant challenges in professional workflows.

By Aman Tyagi, Hemanth Boinpally, Jonathan Chen, Douglas Gebert, Steven Hickson

llms agents diffusion multimodal benchmarks

arXiv AI

Aug 13

Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages

arXiv:2608. 12278v1 Announce Type: cross Abstract: Artificial intelligence tools for education and language support are increasingly framed as scalable responses to access gaps in under-resourced communities.

By Avijit Roy, Proma Roy

nlp benchmarks

arXiv AI

Aug 13

SteeringSafety: Benchmarking Representation Steering in LLMs Across Safety Perspectives

arXiv:2509. 13450v3 Announce Type: replace Abstract: We introduce SteeringSafety, a benchmark for evaluating representation steering methods across nine safety perspectives spanning 18 datasets.

By Vincent Siu, Nicholas Crispino, David Park, Nathan W. Henry, Zhun Wang, Yang Liu, Dawn Song, Chenguang Wang

llms benchmarks safety

arXiv AI

Aug 13

Probably Approximately Correct Maximum A Posteriori Inference

arXiv:2601. 16083v2 Announce Type: replace-cross Abstract: Computing the conditional mode of a distribution, better known as the maximum a posteriori (MAP) assignment, is a fundamental task in probabilistic inference.

By Matthew Shorvon, Frederik Mallmann-Trenn, David S. Watson

benchmarks

arXiv AI

Aug 13

P2MFDS: A Privacy-Preserving Multimodal Fall Detection System for Elderly People in Bathroom Environments

arXiv:2506. 17332v2 Announce Type: replace-cross Abstract: By 2050, people aged 65 and over are projected to make up 16% of the global population.

By Haitian Wang, Yiren Wang, Xinyu Wang, Yumeng Miao, Yuliang Zhang, Yu Zhang, Atif Mansoor

multimodal benchmarks safety

Get the daily briefing