Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,270 stories · RSS feed

arXiv AI

Jul 7

Steering Optimisation Trajectories in Diffusion Representation Learning

arXiv:2607. 05319v1 Announce Type: cross Abstract: We study why diffusion autoencoders can achieve similar image quality while learning substantially different latent structures.

By Rajat Rasal, Avinash Kori, Tian Xia, Ben Glocker

diffusion computer-vision benchmarks

arXiv Machine Learning

Jul 7

BioSEN: A Bio-acoustic Signal Enhancement Network for Animal Vocalizations

arXiv:2605. 12534v3 Announce Type: replace-cross Abstract: Most work in audio enhancement targets human speech, while bioacoustics is less studied due to noisy recordings and the distinct traits of animal sounds.

By Tianyu Song, Ton Viet Ta, Ngamta Thamwattana, Hisako Nomura, Linh Thi Hoai Nguyen

benchmarks

arXiv Machine Learning

Jul 7

Pathways of Visual Information Flow in Vision-Language Models

arXiv:2607. 03358v1 Announce Type: cross Abstract: We study how visual information is routed in vision-language models (VLMs).

By Israfel Salazar, Stella Frank, Dan Oneata, Desmond Elliott, Constanza Fierro

llms multimodal benchmarks

arXiv Machine Learning

Jul 7

Measuring What Matters: A Unified Evaluation Framework for GNN Explainability

arXiv:2607. 04600v1 Announce Type: new Abstract: Graph eXplainable AI (G-XAI) is increasingly important for making Graph Neural Networks interpretable and accountable.

By Francesco Paolo Nerini, Mirko Zaffaroni, Paolo Baracco, Gabriele Ciravegna, Alan Perotti

benchmarks

arXiv Machine Learning

Jul 7

CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learning

arXiv:2607. 03903v1 Announce Type: new Abstract: Multi-task offline safe reinforcement learning (RL) promises to learn a shared optimal safe policy from offline data across multiple tasks.

By Jiayi Guan, Tianle Zhang, Li Shen, Ruiqi Zhang, Ao Zhou, Lusong Li, Guai Chen, Mengjie Li, Alois Knoll, Xiaodong He, Changjun Jiang

llms diffusion reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 7

CLARITY: Medical World Model for Guiding Treatment Decisions by Modeling Context-Aware Disease Trajectories in Latent Space

arXiv:2512. 08029v3 Announce Type: replace Abstract: Clinical decision-making in oncology requires predicting dynamic disease evolution, a task current static AI predictors cannot perform.

By Tianxingjian Ding, Yuanhao Zou, Chen Chen, Mubarak Shah, Yu Tian

llms diffusion benchmarks

arXiv AI

Jul 7

Phase-Preserving Trimodal Transformer for Tropical Forest Biomass Estimation Using Optical and PolInSAR Data

arXiv:2607. 03663v1 Announce Type: cross Abstract: The accurate estimation of Above-Ground Biomass (AGB) in mature tropical forests remains a critical challenge in remote sensing, primarily due to the saturation of Synthetic Aperture Radar (SAR) signals in high-density areas and persistent cloud cover affecting optical imagery.

By Luiz Felipe Parente Santiago (Institute of Computing, Brazilian Army Research Institute in the Amazon), Rosiane Rodrigues de Freitas (Institute of Computing), Daniel Rodrigues dos Santos (Military Institute of Engineering), Felipe Ferrari (Military Institute of Engineering)

llms computer-vision fine-tuning benchmarks

arXiv Machine Learning

Jul 7

MANCE: Manifold Aware Concept Erasure

arXiv:2607. 03973v1 Announce Type: new Abstract: Concept erasure aims to remove a target concept from a representation while preserving the other information encoded in it.

By Matan Avitan, Yoav Goldberg, Yanai Elazar

llms nlp benchmarks

arXiv Machine Learning

Jul 7

GeoFlow: Geo-Aware Modeling of Inter-Area Relationships in Origin-Destination Flow Prediction and Generation

arXiv:2607. 05257v1 Announce Type: new Abstract: Origin-destination (OD) flow modeling underpins urban planning and mobility analysis, but prevailing graph-based methods often neglect salient geographic attributes, limiting their ability to model long-range and multi-area dependencies.

By Zherui Huang, Guanjie Zheng, Hao Xue, Linghe Kong

diffusion benchmarks

arXiv Machine Learning

Jul 7

Air-Plan: Query-Optimized Topology Selection for Over-the-Air Decentralized Federated Learning

arXiv:2607. 04254v1 Announce Type: cross Abstract: Over-the-air (OTA) aggregation exploits the superposition property of wireless multiple-access channels to aggregate model updates from multiple devices within a single transmission slot, significantly reducing communication latency.

By Kaushal Attaluri, Rebeca P. Diaz-Redondo, Manuel Fernandez Veiga

benchmarks safety

arXiv AI

Jul 7

PDFBench: A Benchmark for De novo Protein Design from Function

arXiv:2505. 20346v3 Announce Type: replace-cross Abstract: Function-guided protein design is a crucial task with significant applications in drug discovery and enzyme engineering.

By Jiahao Kuang, Nuowei Liu, Jie Wang, Changzhi Sun, Tao Ji, Yuanbin Wu

benchmarks

arXiv AI

Jul 7

HVR-Met: A Hypothesis-Verification-Replanning Agentic System for Extreme Weather Diagnosis

arXiv:2603. 01121v2 Announce Type: replace Abstract: While deep learning-based weather forecasting paradigms have made significant strides, addressing extreme weather diagnostics remains a formidable challenge.

By Shuo Tang, Jiadong Zhang, Gengxian Zhou, Qizhao Jin, Qinxuan Wang, Yi Hu, Ning Hu, Hongchang Ren, Lingli He, Shiming Xiang, Jingtao Ding, Jian Xu, Jiaolan Fu, Cheng-Lin Liu

agents benchmarks

arXiv AI

Jul 7

AutoCedar: An Agentic Framework for Verifier-Guided Access Control Policy Synthesis

arXiv:2607. 03656v1 Announce Type: cross Abstract: Large Language Models are increasingly used to turn natural-language requirements into code.

By Adarsh Vatsa, Sachi Shome, Yingming Zhou, William Eiers

llms agents benchmarks

arXiv AI

Jul 7

Moonstone: A Multimodal Foundation Model and Benchmark for Lunar Remote Sensing

arXiv:2607. 03644v1 Announce Type: cross Abstract: Decades of orbital missions have produced multi-modal remote sensing data for the Moon, spanning optical imagery, spectroscopy, thermal emission, radar, gravity, and elemental composition.

By Ayush Prasad, Swarnalee Mazumder

llms computer-vision multimodal benchmarks

arXiv AI

Jul 7

Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

arXiv:2607. 03784v1 Announce Type: cross Abstract: While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size.

By Zhenfeng Su, Kang Zhao, Han Bao, Tao Yuan, Zhongzhe Hu, Xianzhi Yu, Wenxuan Wang

llms efficiency benchmarks

arXiv AI

Jul 7

Responsibility Distribution Estimation in Ego-View Accident Videos with Multimodal Large Language Models

arXiv:2607. 03591v1 Announce Type: cross Abstract: Recent studies on multimodal traffic accident understanding have mainly relied on infrastructure-camera footage, satellite imagery, or structured crash records.

By Ryosei Tamura, Andrew Shin

llms agents computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jul 7

TabQueryBench: A Query-Centric Benchmark for Synthetic Tabular Data

arXiv:2607. 03926v1 Announce Type: cross Abstract: Synthetic tabular data support use cases like data sharing, model development under access restrictions, and rapid prototyping of analytical workflows.

By Jialin Zhang, Fenghao Dong, Yajie Zhou, Vyas Sekar, Shinan Liu

benchmarks safety data-engineering

arXiv AI

Jul 7

Harness-Aware Self-Evolving: Co-Evolving Model Weights, Harness, and Task Solutions

arXiv:2607. 03935v1 Announce Type: new Abstract: Self-evolving frameworks usually optimize task solutions while treating the surrounding harness as fixed.

By Haochen Luo, Yi Huang, Sichun Luo, Fengyuan Liu, Lei Li, Zefa Hu, Junlan Feng, Qi Liu

llms agents benchmarks

arXiv AI

Jul 7

MentalThink: Shaping Thoughts in Mental SVG World

arXiv:2607. 03530v1 Announce Type: new Abstract: We introduce MentalThink, a visual-symbolic reasoning paradigm that equips Multimodal LLMs (MLLMs) with an executable mechanism for "mental" visualization.

By Kangheng Lin, Jisheng Yin, Dingming Li, En Yu, Yana Wei, Han Zhou, Liang Zhao, Hongyu Zhou, Hongbo Peng, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Jingyu Wang

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jul 7

SWIFT: Spatio-temporal Wavelet Integrated Forecasting Framework for Workload Traces

arXiv:2607. 02524v1 Announce Type: cross Abstract: Accurate cloud workload forecasting is pivotal for efficient resource management but remains challenging as workloads are highly volatile and prone to sudden bursts.

By Zeyuan Ding, Lingfeng Zheng, Dian Ding, Guangtao Xue

benchmarks

Get the daily briefing