Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,805 stories · RSS feed

arXiv AI

Jun 2

CoCoVideo: The High-Quality Commercial-Model-Based Contrastive Benchmark for AI-Generated Video Detection

arXiv:2606. 00101v1 Announce Type: cross Abstract: With the rapid advancement of artificial intelligence generated content (AIGC) technologies, video forgery has become increasingly prevalent, posing new challenges to public discourse and societal security.

By Huidong Feng, Wentao Chen, Jie Chen, Xinqi Cai, Ruolong Ma, Yinglin Zheng, Yuxin Lin, Ming Zeng

llms diffusion multimodal benchmarks

arXiv AI

Jun 2

3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code

arXiv:2606. 01057v1 Announce Type: cross Abstract: Procedural 3D modeling through code is emerging as a versatile paradigm, offering deterministic, engine-ready, and precisely editable assets that neural 3D generators inherently lack.

By Yipeng Gao, Lei Shu, Genzhi Ye, Xi Xiong, Ameesh Makadia, Meiqi Guo, Laurent Itti, Jindong Chen

llms agents multimodal benchmarks

arXiv AI

Jun 2

TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models

arXiv:2606. 00023v1 Announce Type: cross Abstract: The rapid development of Language Diffusion Models (LDMs) challenges the dominant position of auto-regressive competitors in language processing.

By Yichuan Mo, Yukun Jiang, Yanbo Shi, Mingjie Li, Michael Backes, Yang Zhang, Yisen Wang

diffusion benchmarks safety

arXiv Machine Learning

Jun 2

Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies

arXiv:2606. 01151v1 Announce Type: new Abstract: Behavior cloning with high-capacity generative policies achieves strong imitation performance, but is often limited by demonstration coverage and distribution shift.

By Hikmet Simsir, Ozgur S. Oguz

diffusion reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 2

V2I Work Zone Geometry Reconstruction with Pose-Conditioned UWB Range Denoising

arXiv:2606. 00119v1 Announce Type: cross Abstract: Reliable work zone mapping is important for connected and autonomous vehicles (CAVs) to navigate safely and smoothly through work zone areas.

By Jiaxi Liu, Hangyu Li, Yang Cheng, Rui Gana, Junwei You, Weizhe Tang, Peng Zhang, Steven T. Parker, Xiaopeng Li, Bin Ran

agents benchmarks

arXiv AI

Jun 2

DLLM-JEPA: Joint Embedding Predictive Architectures for Masked Diffusion Language Models

arXiv:2606. 00091v1 Announce Type: cross Abstract: Joint Embedding Predictive Architectures (JEPAs) have reshaped self-supervised representation learning in vision.

By Sangdae Nam

llms rag diffusion fine-tuning benchmarks

arXiv AI

Jun 2

CodeCytos: AI-assisted spatial molecular imaging analysis via code-augmented agent action space

arXiv:2606. 00472v1 Announce Type: cross Abstract: Conventional tissue image analysis software provides foundational capabilities for cellular analysis, including segmentation, basic morphological feature extraction, and spatial organization analysis.

By Hung Q. Vo, Huy Q. Vo, Son T. Ly, Zhihao Wan, Anh-Vu Nguyen, Hong Zhao, Jianting Sheng, Stephen T. C. Wong, Hien V. Nguyen

llms agents computer-vision benchmarks

arXiv AI

Jun 2

Multi-Agent Conformal Prediction with Personalized Statistical Validity

arXiv:2606. 00717v1 Announce Type: cross Abstract: Uncertainty quantification is essential in high-stakes machine learning tasks.

By Martin V. Vejling, Christophe A. N. Biscio, Adrien Mazoyer, Petar Popovski, Shashi Raj Pandey

agents benchmarks safety

arXiv AI

Jun 2

Silent Failures in Physical AI: A Literature Review of Runtime Action Authorization for Autonomous Systems

arXiv:2606. 00090v1 Announce Type: cross Abstract: Physical AI systems increasingly map multimodal observations, language instructions, and learned world representations into physically consequential actions.

By Barak Or

agents robotics multimodal benchmarks safety

arXiv AI

Jun 2

Truth, Trust, and Trouble: Medical AI on the Edge

arXiv:2507. 02983v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) hold significant promise for transforming digital health by enabling automated medical question answering.

By Mohammad Anas Azeez, Rafiq Ali, Ebad Shabbir, Zohaib Hasan Siddiqui, Gautam Siddharth Kashyap, Jiechao Gao, Usman Naseem

llms nlp benchmarks safety

arXiv AI

Jun 2

Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation

arXiv:2606. 01725v1 Announce Type: new Abstract: Agentic AI completes tasks through iterative planning, tool use, and reasoning based on observed outcomes.

By Donghwan Kim, Prakhar Singh, Younghoon Min, Jongryool Kim, Jongse Park, Kiwan Maeng

llms agents benchmarks

arXiv AI

Jun 2

On the Difficulty of Learning a Meta-network for Training Data Selection

arXiv:2606. 00571v1 Announce Type: cross Abstract: Synthetic data are increasingly used to train neural networks, yet distributional mismatch with real data limits their effectiveness when used indiscriminately.

By Zilin Du, Junqi Zhao, Boyang Albert Li

benchmarks data-engineering

arXiv AI

Jun 2

When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystems

arXiv:2606. 00448v1 Announce Type: cross Abstract: LLM agents increasingly rely on community-contributed skills that expand an agent's operational capability set.

By Su Wang, Pin Qian, Yihang Chen, Junxian You, Xiaoyuan Wang, Xiaochong Jiang, Lifei Liu, Haoran Yu, Jingzhou Xu

llms agents benchmarks safety

arXiv AI

Jun 2

Fixed Budget is No Harder Than Fixed Confidence in Best-Arm Identification up to Logarithmic Factors

arXiv:2602. 03972v3 Announce Type: replace-cross Abstract: The best-arm identification (BAI) problem is one of the most fundamental problems in interactive machine learning, which has two flavors: the fixed-budget setting (FB) and the fixed-confidence setting (FC).

By Kapilan Balagopalan, Yinan Li, Yao Zhao, Tuan Nguyen, Anton Daitche, Houssam Nassif, Kwang-Sung Jun

benchmarks

arXiv AI

Jun 2

AgentxGCore: Agentic AI for Next-Generation Mobile Core Network

arXiv:2606. 00417v1 Announce Type: cross Abstract: To meet the stringent requirements of emerging applications and the increasingly complex network management and operation, the Next Generation Mobile Networks (NextG), or 6G, will adopt an AI-native architecture on the Core Network (CN).

By Maria Katarine Santana Barbosa, Kelvin L. Dias

llms agents benchmarks

arXiv AI

Jun 2

DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs

arXiv:2606. 01024v1 Announce Type: cross Abstract: Discrete Masked diffusion language models generate text by iterative parallel decoding, but few-step decoding suffers from a tradeoff between length and quality: with a fixed step budget, standard methods can generate a short, high-quality output, or they can produce long but repetitive text.

By Longxuan Yu, Yunshu Wu, Yu Fu, Siheng Xiong, Rob Brekelmans, Hui Liu, Yue Dong, Greg Ver Steeg

llms rag diffusion nlp benchmarks

arXiv AI

Jun 2

AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

arXiv:2507. 08038v3 Announce Type: replace-cross Abstract: Language model agents are increasingly used to automate scientific research, yet evaluating their scientific contributions remains a challenge.

By Talor Abramovich, Gal Chechik

llms agents benchmarks

arXiv AI

Jun 2

Cross-Axis Feature Fusion with Joint-Wise Motion Difference Prediction for Text-Based 3D Human Motion Editing

arXiv:2606. 01014v1 Announce Type: cross Abstract: We address text-based 3D human motion editing, where the goal is to preserve the style and structure of a source motion while applying edits described in natural language.

By Gyojin Han, Junmo Kim

llms diffusion benchmarks safety

arXiv AI

Jun 2

PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

arXiv:2606. 01016v1 Announce Type: cross Abstract: While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of simple transcription.

By Sicheng Yang, Shulan Ruan, Shiwei Wu, Yu Liu, Lu Fan, Zhi Li, You He

llms multimodal benchmarks safety

arXiv AI

Jun 2

LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

arXiv:2602. 16902v4 Announce Type: replace Abstract: We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs).

By Juliusz Ziomek, William Bankes, Lorenz Wolf, Shyam Sundhar Ramesh, Xiaohang Tang, Ilija Bogunovic

llms benchmarks

Get the daily briefing