arXiv AI

SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation

arXiv:2607. 07469v1 Announce Type: cross Abstract: Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages.

arXiv AI

Jun 11

A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data

arXiv:2601. 17717v3 Announce Type: replace Abstract: Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities.

By Kaituo Zhang, Mingzhi Hu, Hoang Anh Duy Le, Fariha Kabir Torsha, Zhimeng Jiang, Minh Khai Bui, Chia-Yuan Chang, Yu-Neng Chuang, Zhen Xiong, Ying Lin, Guanchu Wang, Na Zou

llms

arXiv Machine Learning

1d ago

Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

arXiv:2410. 13341v4 Announce Type: replace Abstract: High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem.

By Florian E. Dorner, Vivian Y. Nastl, Moritz Hardt

llms

arXiv AI

Jul 24

RE-AD: Real-Time Requirement Adherence for Data Labeling

arXiv:2607. 20455v1 Announce Type: cross Abstract: Human-annotated data remains fundamental to training frontier Large Language Models (LLMs).

By Siddarth Malreddy, Ishan Nigam, Akshay Arora, Nikhil Mittal, Subrat Sahu

llms benchmarks

arXiv AI

Jun 3

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

arXiv:2606. 03650v1 Announce Type: cross Abstract: Choosing or ranking language models for a specific application is hardest when no task-specific labeled data exists, and standard public benchmarks cannot be trusted, their items having likely leaked into pretraining, so scores reflect memorization rather than fitness.

By Alexander Apartsin, Yehudit Aperstein

llms benchmarks safety

arXiv Machine Learning

Jun 2

How to Correctly Report LLM-as-a-Judge Evaluations

arXiv:2511. 21140v4 Announce Type: replace Abstract: Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators.

By Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy-yong Sohn, Kangwook Lee

llms safety

arXiv AI

Aug 5

Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling

arXiv:2605. 13801v2 Announce Type: replace-cross Abstract: As generative AI models such as large language models (LLMs) become more pervasive, ensuring the safety, robustness, and overall trustworthiness of these systems is paramount.

By Deepak Pandita, Flip Korn, Chris Welty, Christopher M. Homan

llms safety

arXiv AI

Jun 24

AI-PAVE-Br: Leveraging Large Language Models for Enhanced Product Attribute Value Extraction through a Golden Set Approach

arXiv:2606. 24655v1 Announce Type: cross Abstract: The explosive growth and complexity of product data within the dynamic Brazilian e-commerce landscape demand robust and specialized methods for structured information extraction.

By Murilo Gazzola, Hugo Gobato Souto, Samuel Silva, J\'ulia Schubert Peixoto, Felipe Siqueira, Andr\'e Luis Pedroso de Morais, Caio Gomes

llms nlp benchmarks

arXiv Machine Learning

Jun 5

Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

arXiv:2606. 05308v1 Announce Type: new Abstract: With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set.

By Abhishek Divekar

llms benchmarks safety

arXiv AI

Jun 16

Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability

arXiv:2606. 15029v1 Announce Type: new Abstract: LLM judges are used to reduce the need for costly human labor in evaluating open-ended text generation.

By Alyssa Unell, Natalie Dullerud, Naomi Boneh, Meena Jagadeesan, Tatsu Hashimoto, Nigam Shah, Sanmi Koyejo

llms safety

arXiv AI

Jun 16

CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment

arXiv:2606. 15396v1 Announce Type: cross Abstract: Malicious content generated from large language models (LLMs) could pose severe safety risks and ethical concerns.

By Wenbo Yu, Bohua Wang, Hao Fang, Kuofeng Gao, Jingru Zeng, Xiaochen Yang, Tianyi Zhang, Xiaoxiao Ma, Jiawei Kong, Hao Wu, Bin Chen, Shu-Tao Xia, Min Zhang

llms rag benchmarks safety

arXiv Machine Learning

Jun 15

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

arXiv:2606. 13221v2 Announce Type: replace Abstract: Evaluating new large language models typically requires costly human annotation campaigns at scale.

By Bora Kargi, David Salinas

llms safety

arXiv AI

Jul 28

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.

By Yunting Song, Matthew Watson, Peter Grabowski, Jun Qin

llms reinforcement-learning benchmarks safety data-engineering

Related stories