Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,071 stories · RSS feed

arXiv AI

Jun 24

AI-PAVE-Br: Leveraging Large Language Models for Enhanced Product Attribute Value Extraction through a Golden Set Approach

arXiv:2606. 24655v1 Announce Type: cross Abstract: The explosive growth and complexity of product data within the dynamic Brazilian e-commerce landscape demand robust and specialized methods for structured information extraction.

By Murilo Gazzola, Hugo Gobato Souto, Samuel Silva, J\'ulia Schubert Peixoto, Felipe Siqueira, Andr\'e Luis Pedroso de Morais, Caio Gomes

llms nlp benchmarks

arXiv AI

Jun 24

Toward Self-Evolution-Ready Workflow Harnesses: A Reversible Migration Path and Convertibility Taxonomy for Expert LLM Pipelines

arXiv:2606. 24598v1 Announce Type: cross Abstract: While expert-validated "LLM + script" workflows deliver significant value, they remain static: they encode hard-won domain knowledge yet fail to adapt execution based on feedback.

By Yimo Lin, Zhen Zhang, Yibin Li

llms agents benchmarks

arXiv AI

Jun 24

Heterogeneous 2D/1D Signal Representation Fusion for Underwater Acoustic Modulation Recognition Under Distribution Shift

arXiv:2606. 23702v1 Announce Type: cross Abstract: Modulation recognition systems rely on heterogeneous signal representations.

By Ronglai Qian, Liang An, Xiaoyan Wang, Qing Fan, Ziwei Huang, Yang Ye

benchmarks

arXiv AI

Jun 24

PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation

arXiv:2606. 24081v1 Announce Type: cross Abstract: As Text-to-Image (T2I) jailbreak techniques evolve rapidly, existing benchmarks and reproduction workflows often struggle to keep pace.

By Leyi Sheng, Han Sun, Zhen Sun, Yuntao Yue, Jinlin Wu, Xinlei He, Jiaheng Wei

agents diffusion multimodal benchmarks safety

arXiv AI

Jun 24

VeriPilot: An LLM-Powered Verilog Debugging Framework

arXiv:2606. 23759v1 Announce Type: cross Abstract: Verilog debugging remains one of the most time-consuming stages in digital circuit design.

By Yihan Wang, Cheng Liu, Jiazheng Zhang, Lei Zhang, Long Cheng, Xiaowei Li, Huawei Li

llms benchmarks

arXiv Machine Learning

Jun 24

An LLM-based Two-Stage Transformer Framework for Cross-Domain Bearing Fault Diagnosis with Limited Data

arXiv:2606. 24459v1 Announce Type: new Abstract: Bearing fault diagnosis faces critical challenges when dataset heterogeneity, operating condition variations, and limited labeled data occur simultaneously in industrial environments.

By Jinghan Wang, Feng Cheng, Wentao Wu, Hang Li, Gaoliang Peng, Tianchen Liu

llms rag fine-tuning benchmarks safety

arXiv Machine Learning

Jun 24

Experiments with Optimal Model Trees

arXiv:2503. 12902v4 Announce Type: replace Abstract: Model trees provide an appealing way to perform interpretable machine learning for both classification and regression problems.

By Sabino Francesco Roselli, Eibe Frank

benchmarks safety

arXiv AI

Jun 24

Entity Resolution via Batched Oracle Queries

arXiv:2606. 24407v1 Announce Type: cross Abstract: We consider an oracle that processes a limited batch of records at a time and clusters those that refer to the same real-world entity.

By Lorenzo Balzotti, Donatella Firmani, Luca Gagliardelli, Giovanni Simonini

benchmarks

arXiv AI

Jun 24

When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models

arXiv:2605. 08245v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input.

By Harshvardhan Saini, Samyak Jha, Yiming Tang, Dianbo Liu

llms rag agents fine-tuning multimodal benchmarks safety

arXiv AI

Jun 24

FP8 is All You Need (Part 2): Efficient Ozaki-Bailey Style FFT Through Tensor-core Garner Reformulation and Kulisch Escape Route

arXiv:2606. 23698v1 Announce Type: cross Abstract: NVIDIA's Blackwell Ultra (B300) cuts FP64 vector throughput to ~1.

By Satoshi Matsuoka

efficiency benchmarks

arXiv AI

Jun 24

CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

arXiv:2409. 11363v2 Announce Type: replace-cross Abstract: AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research.

By Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, Arvind Narayanan

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 24

A Time-Reparameterized Cumulative Intensity Extrapolation Sampler for Discrete Flow Matching

arXiv:2606. 24140v1 Announce Type: new Abstract: Discrete flow matching (DFM) provides a principled framework for generative modeling on discrete state spaces via continuous-time Markov chain dynamics.

By Feiyang Fu, Hehe Fan

diffusion benchmarks

arXiv Machine Learning

Jun 24

RE4: Transformation-aware Imitation of Object Interactions Using Manipulation Modes

arXiv:2606. 24403v1 Announce Type: cross Abstract: Object interaction tasks have been a focus of advances in imitation learning.

By Arsh Chawla, Rahul Shome

diffusion computer-vision robotics benchmarks safety

arXiv AI

Jun 24

Average Rankings Mask Per-Subject Optimality: A Friedman-Nemenyi Benchmark of EEG Motor-Imagery BCI Decoders

arXiv:2606. 24394v1 Announce Type: cross Abstract: Electroencephalography (EEG) is the dominant non-invasive modality for brain-computer interfaces (BCIs), yet reliable decoding of motor imagery is hampered by inter- and intra-individual variability.

By Xavier Vasques, Paul Barbaste, Olivier Oullier

benchmarks

arXiv AI

Jun 24

Offline Reinforcement Learning for Warehouse SLAM Throughput Control

arXiv:2606. 23978v1 Announce Type: cross Abstract: We present an offline reinforcement learning (RL) framework for optimizing SLAM throughput control in a warehouse fulfillment environment.

By Tina Dongxu Li, Mouhacine Benosman, Rajat Kumar, Kevin Tan, Ken Meszaros, Trevor Dardik

reinforcement-learning efficiency benchmarks

arXiv AI

Jun 24

ZONOS2 Technical Report

arXiv:2606. 24320v1 Announce Type: cross Abstract: We present ZONOS2 8B, our latest TTS model, which achieves state-of-the-art naturalness, prosody, and voice cloning fidelity.

By Gabriel Clark, Sofian Mejjoute, Mohamed Osman, George Close, Beren Millidge

efficiency benchmarks

arXiv AI

Jun 24

Probing the Misaligned Thinking Process of Language Models

arXiv:2606. 24251v1 Announce Type: new Abstract: Large language models exhibit a growing range of misaligned behaviors such as strategic deception, sandbagging, and self-preservation.

By Kaiwen Zhou, Constantin Venhoff, Jonathan Michala, Xin Eric Wang, William Saunders

llms benchmarks

arXiv Machine Learning

Jun 24

Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective

arXiv:2512. 10244v2 Announce Type: replace-cross Abstract: Semi-supervised few-shot learning (SSFSL) resembles real-world applications such as auto-annotation, as it aims to learn a model from a few labeled and abundant unlabeled task-specific examples to annotate the unlabeled ones.

By Tian Liu, Anwesha Basu, James Caverlee, Shu Kong

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 24

Multimedia and Visual Analytics in the Agentic Era

arXiv:2504. 06138v3 Announce Type: replace-cross Abstract: Professional users need tools to help them gain actionable insights from large multimedia collections.

By Marcel Worring, Jan Zah\'alka, Stef van den Elzen, Maximilian T. Fischer, Daniel A. Keim

agents computer-vision benchmarks

arXiv AI

Jun 24

Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning

arXiv:2606. 24064v1 Announce Type: new Abstract: Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories, effectively transferring what to answer rather than how to reason.

By Tianyuan Shi, Canbin Huang, Bei Li, Xin Chen, Xiaojun Quan, Jingang Wang, Qifan Wang

llms agents reinforcement-learning fine-tuning efficiency benchmarks

Get the daily briefing