Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,941 stories · RSS feed

arXiv Machine Learning

Jun 8

OPTIMUS-Prime: Minimal and Sufficient Concept Explanations for Deep Vision Models

arXiv:2606. 07180v1 Announce Type: cross Abstract: The growing demand for transparency in automated decision-making has propelled eXplainable Artificial Intelligence (XAI) to the forefront of machine learning research.

By Arthur Hoarau, Chenrui Zhu, Vu Linh Nguyen

computer-vision benchmarks

arXiv AI

Jun 8

Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition

arXiv:2606. 07309v1 Announce Type: cross Abstract: Instruction-following audio language models (ALMs) can be augmented with explicit acoustic cues, yet it remains unclear whether such cues are used in a grounded way when the raw audio is already available.

By Iosif Tsangko, Andreas Triantafyllopoulos, Bj\"orn W. Schuller

llms benchmarks safety

arXiv AI

Jun 8

Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation

arXiv:2606. 07244v1 Announce Type: cross Abstract: Vision-Language Navigation in Continuous Environments (VLN-CE) requires agents to follow natural-language instructions while navigating in real-world-like environments.

By Haoxiang Shi, Xiang Deng, Haoyu Zhang, Qiaohui Chu, Yaowei Wang, Liqiang Nie

agents diffusion multimodal benchmarks

arXiv Machine Learning

Jun 8

Diagnosing Visual Ignorance in Vision-Language Models

arXiv:2606. 06890v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.

By Runyu Zhou, Qi Zhang, Qixun Wang, Yisen Wang

llms multimodal benchmarks

arXiv AI

Jun 8

EvoClaw: Evaluating AI Agents on Continuous Software Evolution

arXiv:2603. 13428v2 Announce Type: replace-cross Abstract: With AI agents increasingly deployed as long-running systems, it becomes essential to autonomously construct and continuously evolve customized software to enable interaction within dynamic environments.

By Gangda Deng, Zhaoling Chen, Zhongming Yu, Haoyang Fan, Yuhong Liu, Yuxin Yang, Dhruv Parikh, Rajgopal Kannan, Le Cong, Mengdi Wang, Qian Zhang, Viktor Prasanna, Xiangru Tang, Xingyao Wang

agents benchmarks

arXiv Machine Learning

Jun 8

On the Effect of Neural Field Reparameterization for 4DVAR

arXiv:2509. 21751v2 Announce Type: replace Abstract: Four-dimensional variational data assimilation (4DVAR) is a cornerstone of numerical weather prediction, yet it remains computationally intensive and sensitive to initialization due to the non-convexity of its objective function.

By Jaemin Oh

benchmarks safety

arXiv AI

Jun 8

Beyond the Black Box: Interpretability of Agentic AI Tool Use

arXiv:2605. 06890v3 Announce Type: replace Abstract: AI agents are promising for high-stakes enterprise workflows, but dependable deployment remains limited because tool-use failures are difficult to diagnose and control.

By Hariom Tatsat, Ariye Shater

llms agents benchmarks safety

arXiv Machine Learning

Jun 8

Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns

arXiv:2503. 03660v4 Announce Type: replace Abstract: We introduce a sequence-conditioned critic for Soft Actor-Critic (SAC) that models trajectory context with a lightweight Transformer and trains on aggregated $N$-step targets.

By Dong Tian, Onur Celik, Gerhard Neumann

llms rag reinforcement-learning benchmarks

arXiv Machine Learning

Jun 8

SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

arXiv:2606. 07098v1 Announce Type: cross Abstract: We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression.

By Ernests Lavrinovics, Marco Letizia, Roy Janco, Shai Segal, Johannes Bjerva, Maurizio Pierini

llms benchmarks

arXiv Machine Learning

Jun 8

TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation

arXiv:2606. 07053v1 Announce Type: cross Abstract: Pose-guided text-to-image generation often suffers from limb distortions and feature crosstalk in complex multi-person scenarios.

By Dian Gu, Zhengyi Yang

llms diffusion multimodal benchmarks safety

arXiv AI

Jun 8

FreeAnimate: Training-Free Human Image Animation with Preview-Guided Denoising

arXiv:2606. 06885v1 Announce Type: cross Abstract: Human Image Animation has seen significant advancements, primarily driven by diffusion models.

By Yuan Zeng, Yujia Shi, Zongqing Lu, QingMin Liao

diffusion benchmarks safety

arXiv AI

Jun 8

Graph Neural Network leveraging Higher-order Class Label Connectivity for Heterophilous Graphs

arXiv:2606. 07475v1 Announce Type: cross Abstract: Node classification in graph neural networks (GNNs) has been widely applied in various fields of graph analysis.

By Takuto Takahashi, Itsuki Nakayama, Takahiro Mitani, Ryosuke Kikuchi, Yuya Sasaki, Makoto Onizuka

rag benchmarks

arXiv AI

Jun 8

DualGate-Net: A Prior-Gated Dual-Encoder Framework for Histopathology Cell Detection

arXiv:2606. 07222v1 Announce Type: cross Abstract: Cell detection in histopathology images strongly depends on surrounding tissue context, where visually similar cells may belong to different classes under different microenvironments.

By Bahman Jafari Tabaghsar, Son Tran, K. Devaraja, Atul Sajjanhar

benchmarks

arXiv AI

Jun 8

Neuro-Symbolic Learning for Long-Horizon Task Planning Under Complex Logical Constraints

arXiv:2606. 06877v1 Announce Type: cross Abstract: Task planning often suffers from severe efficiency bottlenecks when robots must reason over long-horizon action sequences under complex logical constraints, including object affordances, spatial relationships, and sequential action dependencies.

By Qiwei Du, Zitong Zhan, Shaoshu Su, Bowen Li, Yi Du, Zhipeng Zhao, Taimeng Fu, Sebastian Scherer, Jiaoyang Li, Chen Wang

robotics benchmarks safety

arXiv AI

Jun 8

DxPTA: An Architecture Design Space Exploration with Optical Dataflow-guided Strategy for HW/SW Co-Design of Photonic Transformer Accelerators

arXiv:2606. 06515v1 Announce Type: cross Abstract: Transformer-based networks have emerged as prominent AI models with state-of-the-art performance, which potentially pave the way toward artificial general intelligence (AGI).

By Rachmad Vidya Wicaksana Putra, Solomon Micheal Serunjogi, Mahmoud Rasras, Muhammad Shafique

llms benchmarks

arXiv AI

Jun 8

ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning

arXiv:2606. 06915v1 Announce Type: cross Abstract: Test-time compute (TTC) scaling has emerged as a powerful paradigm for improving large language model (LLM) reasoning by allocating additional compute during inference, e.

By Vladislav Smirnov (MBZUAI), Chieu Nguyen (MBZUAI), Sergey Senichev (Independent Researcher), Minh Ngoc Ta (MBZUAI), Ekaterina Fadeeva (ETH Z\"urich), Artem Vazhentsev (MBZUAI), Daria Galimzianova (MBZUAI), Nikolai Rozanov (MBZUAI, Imperial College London), Viktor Mazanov (Innopolis University), Jingwei Ni (ETH Z\"urich), Tianyi Wu (NUS), Igor Kiselev (Accenture), Mrinmaya Sachan (ETH Z\"urich), Iryna Gurevych (MBZUAI), Preslav Nakov (MBZUAI), Timothy Baldwin (MBZUAI), Artem Shelmanov (MBZUAI)

llms rag benchmarks

arXiv AI

Jun 8

Towards Efficient and Exact Forgetting Services in Pre-Trained-Model-based Continual Learning

arXiv:2505. 12239v2 Announce Type: replace-cross Abstract: In Continual Learning (CL), using a Pre-Trained Model (PTM) as the feature extractor has become a popular practice.

By Yajiang Huang, Jianheng Tang, Kejia Fan, Huiping Zhuang, Anfeng Liu, Tian Wang, Yunhuai Liu, Mianxiong Dong, Houbing Herbert Song

benchmarks safety

arXiv AI

Jun 8

DyCon: Dynamic Reasoning Control via Evolving Difficulty Modeling

arXiv:2606. 07108v1 Announce Type: new Abstract: Recent advances in Large Reasoning Models (LRMs) demonstrate remarkable performance improvements by iteratively reflecting, exploring, and executing complex tasks, yet suffer from inefficiencies due to redundant reasoning, known as "overthinking".

By Tengyao Tu, Yulin Li, Hui-Ling Zhen, Libo Qin, Zhoujun Wei, Jinghua Piao, Zhuotao Tian, Yong Li, Min Zhang

rag nlp benchmarks

arXiv AI

Jun 8

Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation

arXiv:2606. 06836v1 Announce Type: cross Abstract: Language-guided UAV agents must execute long-horizon semantic instructions while producing smooth, physically feasible continuous flight commands, yet existing Vision-Language Navigation (VLN) benchmarks typically use discrete or coarse actions and existing UAV Vision-Language-Action (VLA) tasks focus on short, atomic maneuvers.

By Xiangyi Zheng, Xiangyu Wang, Qinan Liao, Zimu Tang, Yue Liao, Dongyue Lyu, Guodong Wang, Junjie Liu, Si Liu

llms agents diffusion multimodal benchmarks

arXiv Machine Learning

Jun 8

Breaking the Tuning Barrier: Zero-Hyperparameters Yield Multi-Corner Analysis Via Learned Priors

arXiv:2603. 13092v2 Announce Type: replace Abstract: Yield Multi-Corner Analysis validates circuits across 25+ Process-Voltage-Temperature corners, resulting in a combinatorial simulation cost of $O(K \times N)$ where $K$ denotes corners and $N$ exceeds $10^4$ samples per corner.

By Wei W. Xing, Kaiqi Huang, Jiazhan Liu, Hong Qiu, Shan Shen

llms benchmarks

Get the daily briefing