Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,257 stories · RSS feed

arXiv AI

Jul 7

HCSU: A Dataset and Benchmark for Fine-Grained Historical Calligraphy Style Understanding

arXiv:2607. 04147v1 Announce Type: cross Abstract: Automated fine-grained perception of calligraphy styles--a task vital to cultural heritage preservation--remains a critical challenge for Large Vision-Language Models (LVLMs), largely constrained by existing datasets that suffer from modal mixture and flattened labels.

By Yinsheng Yao, Yan Liu, Chen Ye

llms multimodal benchmarks

arXiv AI

Jul 7

Attention Dynamics in Diffusion Models: A Visual Analytics Framework for Human-AI Collaboration

arXiv:2607. 02563v1 Announce Type: cross Abstract: Diffusion-based text-to-image models can synthesize complex and highly structured visual content, yet the emergence and evolution of semantic structure remain difficult to interpret.

By Yiran Xiao, George Legrady

diffusion benchmarks

arXiv AI

Jul 7

A Fair Benchmarking of Deep Relational Database Learning Models

arXiv:2607. 03659v1 Announce Type: cross Abstract: Relational databases (RDBs) are the primary data infrastructure in many enterprises, yet recent deep learning methods designed for RDBs have been evaluated under inconsistent experimental protocols, making fair comparison difficult.

By Kazi F. Akhter, Bharath Ajendla, Manar D. Samad

llms benchmarks

arXiv Machine Learning

Jul 7

Evaluating Time Series Foundation Models for Electricity Price Forecasting: Contamination Risk, Distributional Shifts, and Covariate Dependence

arXiv:2607. 02623v1 Announce Type: new Abstract: Time series foundation models (TSFMs) have shown strong zero-shot forecasting performance, but their generalization in covariate-driven, non-stationary settings is underexplored.

By Zhenghua Pan, Ahmed Aziz Ezzat

benchmarks

arXiv Machine Learning

Jul 7

Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets

arXiv:2605. 17758v2 Announce Type: replace Abstract: Synthetic data is widely used in healthcare to create datasets that preserve statistical properties of real data without exposing sensitive patient information.

By Nitish Nagesh, Pengbao Zhou, Atchuth Naveen Chilaparasetti, Yajat Nagaraj Kiran, Tu Nguyen, Arshia Harish Puthran, Muhjaazee Love, Aadi Sharma, Mahdi Bagheri, Ian Harris, Amir M. Rahmani

llms agents diffusion benchmarks safety

arXiv Machine Learning

Jul 7

TSP with Predictions: Heatmap to Tour with Provable Guarantees

arXiv:2607. 03791v1 Announce Type: cross Abstract: The Traveling Salesperson Problem (TSP) has long served as a benchmark for evaluating the strength of optimization techniques in the classical theory of algorithms.

By Marek Eli\'a\v{s}, Fabrizio Grandoni, Adam Polak, Eleonora Vercesi

benchmarks

arXiv AI

Jul 7

CanniUplift: A Holistic Framework for Mitigating Seller and Incentive Cannibalization in E-commerce Uplift Modeling

arXiv:2607. 05242v1 Announce Type: cross Abstract: Personalized incentive allocation is vital for e-commerce, where uplift modeling is the standard for estimating Individual Treatment Effects (ITE).

By Zuwang He, Shihao Shu, Yuli Qu, Hanyu Gao, Ziliang Zhang, Diwei Chen, Xiangda Yan, Buyu Gao, Tanchao Zhu, Yumeng Li, Junxiong Zhu

benchmarks safety

arXiv Machine Learning

Jul 7

Rethinking AI-Generated Text Detection: A Strong Baseline and the Distribution-Shift Problem That Remains

arXiv:2607. 03680v1 Announce Type: new Abstract: Recent AI-generated text detection work often introduces a new benchmark together with a specialized detector tailored to it.

By Zhuoer Shen, Mingyi Wang, Shaofeng Zou, Yuheng Bu

fine-tuning benchmarks

arXiv Machine Learning

Jul 7

miMamba: EEG-based Emotion Recognition with Multi-scale Inverted Mamba Models

arXiv:2409. 07589v2 Announce Type: cross Abstract: EEG-based emotion recognition holds significant potential in the field of brain-computer interfaces.

By Xin Zhou, Dawei Huang, Xiaojing Peng, Lijun Yin

benchmarks

arXiv AI

Jul 7

PLACEMEM: Toward a Compute-Aware Memory Plane for Lifelong Agents

arXiv:2607. 04089v1 Announce Type: new Abstract: Lifelong agents need more than larger context windows and better retrieval.

By Sukanta Ganguly

agents benchmarks

arXiv Machine Learning

Jul 7

QiVC-Net: Quantum-Inspired Variational Convolutional Network, with Application to Biosignal Classification

arXiv:2511. 05730v2 Announce Type: replace Abstract: In this paper, a learning framework is introduced which incorporates principles of probabilistic inference, variational optimization, and geometry-preserving operations inspired by quantum transformations.

By Amin Golnari, Jamileh Yousefi, Reza Moheimani, Saeid Sanei

benchmarks

arXiv AI

Jul 7

Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs

arXiv:2607. 04371v1 Announce Type: new Abstract: We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment.

By Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, Shay Aharon, Nir Ailon, Vladimir Anisimov, Omer Ullman Argov, Maor Ashkenazi, Tomer Asida, Nave Assaf, Tomer Bar Natan, Alexander Bukharin, Grzegorz Chlebus, Marcin Chochowski, Eric Chung, Mohammad Dabbah, Carlo del Mundo, Ewa Dobrowolska, Ido Galil, Yaniv Galron, Amnon Geifman, Yonatan Geifman, Izik Golan, Alex Gronskiy, Tomasz Grzegorzek, Netanel Haber, Lior Kadoch, Grzegorz Karch, Tomer Keren, Abhinav Khattar, Amir Klein, Tugrul Konuk, Roi Koren, Daniel Korzekwa, Shaun Kotek, Konstantinos Krommydas, Itay Levy, Ofri Masad, Yoav Miron, Pavlo Molchanov, Shahar Mor, Zach Moshe, Saurav Muralidharan, Najeeb Nabwani, Besmira Nushi, Mostofa Patwary, Omri Puny, Johannes Rausch, Tomer Ronen, Sepehr Sameni, Itamar Schen, Elad Segal, Daniel Serebrenik, Ido Shahaf, Soumye Singhal, Daniil Sorokin, Sharath Turuvekere Sreenivas, Marta Stepniewska-Dziubinska, Ali Taghibakhshi, Nima Tajbakhsh, Oren Tropp, Dor Tzur, Anna Warno, Yi-Fu Wu, Michal Zawalski, Jiaqi Zeng, Yian Zhang, Ran Zilberstein, Amit Zuker, Ran El-Yaniv

llms agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jul 7

Seeing Through WiFi: Lightweight Human Pose Estimation with Dynamic Kernel Attention

arXiv:2607. 03196v1 Announce Type: cross Abstract: WiFi-based human pose estimation (HPE) enables the detection and interpretation of human body positions and movements without the need for wearable devices while preserving individual privacy concerns.

By Toan D. Gian, Van-Dinh Nguyen, Vo Phi Son, Nhan Thanh Nguyen, Dinh Thai Hoang, Diep N. Nguyen, Nguyen Cong Luong, Symeon Chatzinotas

computer-vision benchmarks safety

arXiv AI

Jul 7

PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement

arXiv:2607. 02897v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have shown strong capabilities, but they may memorize private information from web data, raising privacy concerns.

By Xianren Zhang, Delvin Ce Zhang, Dongwon Lee, Suhang Wang

llms multimodal benchmarks safety

arXiv AI

Jul 7

URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment

arXiv:2607. 04688v1 Announce Type: cross Abstract: Synthesis planning aiming to find pathways of reactions for a target molecule is one of the most important and challenging tasks in drug discovery.

By Bogdan Zagribelnyy, Ivan Ilin, Nikita Bondarev, Anton Morgunov, Arkadii Lin, Maksim Kuznetsov, Rim Shayakhmetov, Vladimir Aladinskiy, Alex Aliper, Alex Zhavoronkov

llms benchmarks

arXiv AI

Jul 7

Skill Coverage: A Test Adequacy Metric for Agent Skills

arXiv:2606. 20659v2 Announce Type: replace Abstract: Agent skills encode reusable procedural knowledge for large language model (LLM) agents, and existing benchmarks show that such skills can improve task-level performance.

By Boyin Tan, Xiaowei Huang, Youcheng Sun

llms agents benchmarks

arXiv AI

Jul 7

Towards Generalizable Deepfake Image Detection with Vision Transformers

arXiv:2604. 17376v2 Announce Type: replace-cross Abstract: In today's day and age, we face a challenge in detecting deepfake images because of the fast evolution of modern generative models and the poor generalization capability of existing methods.

By Kaliki V Srinanda, M Manvith Prabhu, Hemanth K Mogilipalem, Jayavarapu S Abhinai, Vaibhav Santhosh, Aryan Herur, Deepu Vijayasenan

llms fine-tuning benchmarks

arXiv AI

Jul 7

CP-WSP: A Declarative CP-SAT Framework for Configurable Multi-Constraint Workforce Scheduling

arXiv:2607. 05177v1 Announce Type: new Abstract: Workforce scheduling is an NP-hard combinatorial optimization problem requiring simultaneous satisfaction of labor regulations, coverage requirements, employee preferences and operational objectives.

By Vipul Patel, Anirudh Deodhar, Dagnachew Birru

benchmarks

arXiv AI

Jul 7

VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning

arXiv:2607. 02927v1 Announce Type: cross Abstract: Video understanding is moving beyond closed-context perception toward open-world evidence exploration, a paradigm formalized as Video Deep Research (VDR).

By Zhenkun Gao, Yicheng Bao, Jinlong Peng, Xueheng Li, Theo Huang, Bangwei Liu, Kunquan Li, Zhenye Gan, Tao Hu, Chengjun Xie, Mingqian Yang, Xuanhua He, Zhizhong Zhang, Xin Tan, Chengjie Wang, Yuan Xie

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

Jul 7

TRACE: Capability-Targeted Agentic Training

arXiv:2604. 05336v2 Announce Type: replace Abstract: Models often fail to complete agentic tasks because they lack core capabilities required by the target environment.

By Hangoo Kang, Tarun Suresh, Jon Saad-Falcon, Azalia Mirhoseini

agents reinforcement-learning fine-tuning benchmarks

Get the daily briefing