arXiv AI

A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data

arXiv:2601. 17717v3 Announce Type: replace Abstract: Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities.

arXiv AI

Aug 3

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

arXiv:2607. 28801v1 Announce Type: cross Abstract: Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples.

By Philipp D. Siedler, Jordan Sassoon

llms agents benchmarks safety

arXiv Machine Learning

Jul 24

DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

arXiv:2607. 20465v1 Announce Type: new Abstract: The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end.

By Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, Qifeng Xia, Sizhe Qiu, Linzhuang Sun, Meiyi Qiang, Zhaoyang Han, Xiaochen Ma, Bohan Zeng, Ruichuan An, Conghui He, Wentao Zhang

llms agents fine-tuning benchmarks safety data-engineering

arXiv AI

Aug 11

PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.

By Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu

llms agents benchmarks safety

arXiv Machine Learning

Jul 31

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

arXiv:2604. 13977v2 Announce Type: replace-cross Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent.

By Joel Niklaus, Atsuki Yamaguchi, Michal \v{S}tef\'anik, Guilherme Penedo, Hynek Kydl\'i\v{c}ek, Elie Bakouch, Lewis Tunstall, Edward Emanuel Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, Thomas Wolf

llms

arXiv AI

Jun 3

TSQAgent: Rating Time Series Data Quality via Dedicated Agentic Reasoning

arXiv:2606. 03629v1 Announce Type: new Abstract: Assessing the quality of time series (TS) data is fundamental yet inherently challenging due to the multifaceted nature of quality dimensions.

By Shunyu Wu, Dan Li, Haozheng Ye, Weibin Feng, Jian Lou, Bo Zhang, Wenjie Feng, Chenjuan Guo, See-Kiong Ng

llms agents benchmarks data-engineering

arXiv AI

Jul 8

Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation

arXiv:2607. 05985v1 Announce Type: new Abstract: This paper presents a black-box evaluation framework to systematically assess the ability of Large Language Models (LLMs) to generate Design Structure Matrices (DSMs) from structured technical documentation.

By Niels Potters, Theo Hofman

llms benchmarks safety

arXiv Machine Learning

Jun 5

Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation

arXiv:2606. 05792v1 Announce Type: cross Abstract: TLA+ has supported industrial verification at companies such as Amazon and Microsoft, yet writing correct TLA+ specifications from natural language still requires time and expertise, which limits adoption.

By Arslan Bisharat, Brian Ortiz, Eric Spencer, Khushboo Bhadauria, TaiNing Wang, George K. Thiruvathukal, Konstantin Laufer, Mohammed Abuhamad

llms safety

arXiv AI

Jul 28

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.

By Yunting Song, Matthew Watson, Peter Grabowski, Jun Qin

llms reinforcement-learning benchmarks safety data-engineering

arXiv AI

Jul 9

SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation

arXiv:2607. 07469v1 Announce Type: cross Abstract: Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages.

By Andrea Scarinci, Virginia Negri, Brayan Impata, Suleiman Khan, Victor Martinez, Marcello Federico

llms fine-tuning benchmarks

arXiv AI

Aug 5

TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation

arXiv:2608. 02975v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated impressive performance in MQM-based translation quality (TQ) evaluation, and recent advances in large reasoning models (LRMs) promise even greater improvements.

By Bhavin Jawade, Cameron R. Wolfe

llms efficiency benchmarks

Hugging Face Trending Papers

Jul 7

Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation

This paper presents a black-box evaluation framework to systematically assess the ability of Large Language Models (LLMs) to generate Design Structure Matrices (DSMs) from structured technical documentation. Motivated by the closed-source nature of current Auto-DSM pipelines, the framework introduces a reproducible methodology that benchmarks generated DSMs (GEN-DSMs) against manually validated ground-truth matrices (GT-DSMs).

llms benchmarks safety

arXiv AI

Jun 8

Auditing Training Data in Domain-adapted LLMs: LoRA-MINT

arXiv:2606. 06946v1 Announce Type: cross Abstract: We present LoRA-MINT, a new methodology for Membership Inference Test (MINT) applied to recent Large Language Models (LLMs) fine-tuned for specific Natural Language Processing (NLP) tasks through Low-Rank Adaptation (LoRA).

By Gonzalo Mancera, Daniel DeAlcala, Aythami Morales, Julian Fierrez, Ruben Tolosana, Francisco Jurado

llms nlp fine-tuning benchmarks

Related stories