arXiv AI
Jul 20
Design-Based Supervised Learning with Noisy Human Labels
arXiv:2607. 15455v1 Announce Type: cross Abstract: Researchers increasingly use automated classifiers to label unstructured data for statistical analysis.
AutoEval Done Right: Using Synthetic Data for Model Evaluation
arXiv:2403. 07008v3 Announce Type: replace-cross Abstract: The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuming.
Related stories
arXiv Machine Learning
Jun 25
Autodata: An agentic data scientist to create high quality synthetic data
arXiv:2606. 25996v1 Announce Type: cross Abstract: We introduce Autodata, a general method that enables AI agents to act as data scientists who build high quality training and evaluation data.
arXiv AI
Jun 9
Deep Active Re-Labeling: Toward Noise-Resilient Annotation Efficiency
arXiv:2606. 08718v1 Announce Type: cross Abstract: While Deep Active Learning (DAL) effectively reduces human annotation costs, its efficacy is constrained by human annotation errors.
arXiv AI
Jun 4
Generative Augmented Inference
arXiv:2604. 14575v2 Announce Type: replace-cross Abstract: Large language models enable inexpensive AI-generated annotations, but using them reliably for causal inference remains challenging.
arXiv AI
Jun 4
Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning
arXiv:2605. 23595v2 Announce Type: replace-cross Abstract: The rapid advancement of machine learning has led to an unprecedented expansion of model ecosystems, making it increasingly difficult to assess the reliability of newly released models on unseen and unlabeled data.
Towards Data Science
Jul 27
Reducing Human Annotation with ML Active Learning
In a world where human time is expensive, learn how to use it only when really necessary The post Reducing Human Annotation with ML Active Learning appeared first on Towards Data Science .
arXiv AI
Jun 12
Valid Inference with Synthetic Data via Task Exchangeability
arXiv:2606. 13629v1 Announce Type: cross Abstract: There is a proliferation of work arguing for the use of synthetic data in scientific research.
OpenAI Blog
Sep 19, 2019
Fine-tuning GPT-2 from human preferences
We’ve fine-tuned the 774M parameter GPT-2 language model using human feedback for various tasks, successfully matching the preferences of the external human labelers, though those preferences did not always match our own. Specifically, for summarization tasks the labelers preferred sentences copied wholesale from the input (we’d only asked them to ensure accuracy), so our models learned to copy.
arXiv AI
Jun 2
ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning
arXiv:2603. 09692v2 Announce Type: replace-cross Abstract: Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs), yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains.
arXiv AI
Jul 9
SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation
arXiv:2607. 07469v1 Announce Type: cross Abstract: Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages.
arXiv Machine Learning
Aug 5
Dynamically Allocating Evaluation Effort for Model Ranking
arXiv:2608. 03437v1 Announce Type: cross Abstract: While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability.
arXiv AI
Jul 24
RE-AD: Real-Time Requirement Adherence for Data Labeling
arXiv:2607. 20455v1 Announce Type: cross Abstract: Human-annotated data remains fundamental to training frontier Large Language Models (LLMs).