Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,273 stories · RSS feed

arXiv AI

Jun 8

InvEvolve: Evolving White-Box Inventory Policies via Large Language Models with Performance Guarantees

arXiv:2605. 00369v4 Announce Type: replace-cross Abstract: We study how large language models can be used to generate inventory policies in online settings with non-stationary demand.

By Chenyu Huang, Jianghao Lin, Zhengyang Tang, Bo Jiang, Ruoqing Jiang, Benyou Wang, Lai Wei

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 8

Accelerating Multi-Objective Bayesian Optimisation via Predictive-Gradient Catalysts

arXiv:2606. 06984v1 Announce Type: new Abstract: This paper presents a general acceleration mechanism for multi-objective Bayesian optimisation (MOBO) that leverages Gaussian process predictive gradients as auxiliary signals.

By Alma Rahat, Tinkle Chugh, Jonathan Fieldsend, Richard Allmendinger

benchmarks

arXiv AI

Jun 8

Towards Unified Song Generation and Singing Voice Conversion with Accompaniment Co-Generation

arXiv:2606. 07015v1 Announce Type: cross Abstract: While song generation and singing voice conversion (SVC) have evolved significantly, they have long been developed isolated: the former lacks zero-shot speaker cloning, while the latter overlooks vocal-accompaniment synergy.

By Ziyu Zhang, Chunyu Qiang, Xiaopeng Wang, Yuxin Guo, Kang Yin, Wenjie Tian, Jingbin Hu, Tianlun Zuo, Zhao Guo, Teng Ma, Yuzhe Liang, Chen Zhang, Lei Xie

llms rag diffusion multimodal benchmarks

arXiv AI

Jun 8

SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Models

arXiv:2606. 06907v1 Announce Type: cross Abstract: Large audio language models (LALMs) extend large language models with an audio encoder and large-scale audio data.

By Seonuk Kim, Yonghyeon Jun, Ju Yeon Kang, Jimin Hong, Yoonhyeong Lee, Nam Soo Kim

llms fine-tuning benchmarks

arXiv AI

Jun 8

Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio

arXiv:2603. 08683v2 Announce Type: replace-cross Abstract: Autoregressive "language" models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to 8-bit audio, leaving open whether such approaches work for practical settings (16/24-bit) and can compete with existing codecs.

By Phillip Long, Zachary Novack, Chris Donahue

nlp benchmarks

arXiv Machine Learning

Jun 8

Hermes: Accelerating Long-Latency Load Requests via Perceptron-Based Off-Chip Load Prediction

arXiv:2209. 00188v4 Announce Type: replace-cross Abstract: Long-latency load requests continue to limit the performance of high-performance processors.

By Rahul Bera, Konstantinos Kanellopoulos, Shankar Balachandran, David Novo, Ataberk Olgun, Mohammad Sadrosadati, Onur Mutlu

benchmarks

arXiv AI

Jun 8

The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

arXiv:2606. 07422v1 Announce Type: cross Abstract: Large language models are increasingly used to answer culturally grounded questions across languages, yet it remains unclear whether local cultural knowledge is better accessed through English or the local language.

By Yang Zhang, Xiao Fei, Amr Mohamed, Sarah Almeida Carneiro, Mersin Konomi, Mingmeng Geng, Ahmed Asaad, Guokan Shang, Michalis Vazirgiannis

llms benchmarks

arXiv Machine Learning

Jun 8

Making the Most of Limited Data: Score-Aware Training for Text-to-Music Generation

arXiv:2606. 07387v1 Announce Type: new Abstract: State-of-the-art text-to-music generation systems rely on massive proprietary datasets and industrial-scale compute, making it impossible to disentangle architectural contributions from resource advantages.

By Yun-Chen Cheng, Tzu-Hung Huang, Chih-Pin Tan

benchmarks safety

arXiv Machine Learning

Jun 8

Product units in gated recurrent units improve nuclear-mass prediction

arXiv:2606. 06866v1 Announce Type: new Abstract: The prediction of masses of atomic nuclei using machine learning can complement theoretical models and advance the exploration of poorly known domains of the nuclear chart.

By Ziyuan Li, Paulo S. A. Freitas, John W. Clark, Babette Dellen

benchmarks

arXiv AI

Jun 8

Graph Neural Network leveraging Higher-order Class Label Connectivity for Heterophilous Graphs

arXiv:2606. 07475v1 Announce Type: cross Abstract: Node classification in graph neural networks (GNNs) has been widely applied in various fields of graph analysis.

By Takuto Takahashi, Itsuki Nakayama, Takahiro Mitani, Ryosuke Kikuchi, Yuya Sasaki, Makoto Onizuka

rag benchmarks

arXiv Machine Learning

Jun 8

$\alpha$-PFN: Fast Entropy Search via In-Context Learning

arXiv:2606. 07134v1 Announce Type: new Abstract: Information-theoretic acquisition functions such as Entropy Search (ES) offer a principled exploration-exploitation framework for Bayesian optimization (BO).

By Herilalaina Rakotoarison, Steven Adriaensen, Tom Viering, Carl Hvarfner, Samuel M\"uller, Frank Hutter, Eytan Bakshy

llms benchmarks

arXiv AI

Jun 8

Beyond Skeletons: Learning Animation Directly from Driving Videos with Same2X Training Strategy

arXiv:2606. 06903v1 Announce Type: cross Abstract: Human image animation aims to generate a video from a static reference image, guided by pose information extracted from a driving video.

By Yuan Zeng, Yujia Shi, Yuhao Yang, Dongxia Liu, Zongqing Lu, Wenming Yang, Qingmin Liao

benchmarks safety

arXiv AI

Jun 8

ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios

arXiv:2602. 16073v2 Announce Type: replace-cross Abstract: Developing autonomous driving systems for complex traffic environments requires balancing multiple objectives, such as avoiding collisions, obeying traffic rules, and making efficient progress.

By Kevin Kai-Chun Chang, Ekin Beyazit, Alberto Sangiovanni-Vincentelli, Tichakorn Wongpiromsarn, Sanjit A. Seshia

agents robotics benchmarks

arXiv Machine Learning

Jun 8

Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies

arXiv:2606. 07492v1 Announce Type: cross Abstract: The ranking of recommendation algorithms is a challenging problem since model performance is sensitive to dataset characteristics such as sparsity, sequential structure, and scale.

By Ekaterina Grishina, Stepan Kuznetsov, Askar Tsyganov, Ilya Ivanov, Daria Korovaitceva, Margarita Rusanova, Uliana Parkina, Alexander Derevyagin, Evgeny Frolov, Sergey Samsonov, Anton Lysenko

efficiency benchmarks

arXiv Machine Learning

Jun 8

The Identity Trap in EEG Foundation Models: A Diagnostic Audit

arXiv:2606. 06647v1 Announce Type: new Abstract: Objective.

By Jun-You Lin, Ying Choon Wu, Tzyy-Ping Jung

fine-tuning benchmarks

arXiv AI

Jun 8

UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding

arXiv:2606. 07167v1 Announce Type: cross Abstract: Meaningful multilingual evaluation must test models in the target language and educational context.

By Ahmer Tabassum, Sarfraz Ahmad, Hasan Iqbal, Owais Aijaz, Momina Ahsan, Preslav Nakov

llms benchmarks

arXiv AI

Jun 8

TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment

arXiv:2606. 07451v1 Announce Type: cross Abstract: Vision-language models such as CLIP are highly useful for diverse tasks due to their shared image-text embedding space.

By Sweta Mahajan, Sukrut Rao, Jiahao Xie, Alexander Koller, Bernt Schiele

llms rag multimodal benchmarks safety

arXiv AI

Jun 8

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

arXiv:2605. 21347v3 Announce Type: replace Abstract: Diagnosing failures in LLM agents remains largely manual.

By Akshay Manglik, Apaar Shanker, Kaustubh Deshpande, Jason Qin, Yash Maurya, Veronica Chatrath, Vijay S. Kalmath, Levi Lentz, Yuan Xue

llms agents benchmarks

Hugging Face Trending Papers

Jun 8

ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China

We introduce ChinaHeritaQA, a multimodal benchmark dataset for evaluating the cultural reasoning abilities of vision-language models (VLMs) on UNESCO World Heritage sites in China. The dataset comprises 2,279 in-the-wild images paired with 14,133 bilingual (Chinese/English) multiple-choice QA pairs spanning seven cognitive dimensions, from basic identity recognition to historical periodization and architectural analysis.

llms nlp multimodal benchmarks

Hugging Face Trending Papers

Jun 8

Order Matters: Unveiling the Hidden Impact of Macro Placement Sequences via Proxy-Guided LLM Evolution

Macro placement is a fundamental step in modern chip physical design, playing a crucial role in determining the solution quality of high-dimensional combinatorial optimization problems. Despite recent advancements in machine learning for spatial coordinate determination, the temporal dimension of placement sequencing remains largely governed by static heuristics.

llms benchmarks

Get the daily briefing