The study explores how diversity-driven sampling can reduce the size of pre‑training datasets for ModernBERT models in French while preserving performance. By comparing several diversity algorithms, the authors pre‑train models on a fraction of the data and evaluate them on multiple French benchmarks. Results show that both random and diversity sampling can cut dataset size by up to 94% and training time by up to 73%, with diversity sampling sometimes improving model quality by up to 10 points.
arXiv:2606. 15216v1 Announce Type: cross Abstract: Diversity plays a critical role in data selection, improving performance under fixed data budgets by reducing redundancy and repetition.
arXiv:2510. 16882v4 Announce Type: replace-cross Abstract: Supervised fine-tuning (SFT) is a commonly used technique to adapt large language models (LLMs) to downstream tasks.
The paper introduces ZipCal, a model‑agnostic data curation method that selects calibration data for post‑training compression of large language models by maximizing lexical diversity using Zipfian power laws. ZipCal outperforms uniform random sampling on pruning benchmarks and matches a state‑of‑the‑art perplexity‑based approach while being roughly 240× faster due to its linear complexity. The authors provide code and experiments at their GitHub repository.
arXiv:2502. 11027v5 Announce Type: replace Abstract: Large language model (LLM) scaling inference is key to unlocking greater performance, and leveraging diversity has proven an effective way to enhance it.
Chopthin-Consensus Power Sampling (CCPS) is a new inference-time decoding method for large language models that uses the Chopthin resampler to preserve diversity among particle trajectories. By enforcing an upper bound on weight ratios instead of equal-weight resampling, CCPS maintains a richer set of distinct reasoning paths and guarantees a lower bound on effective sample size. Coupled with a semantic-majority selection mechanism, CCPS achieves higher oracle coverage and matches or surpasses baseline accuracy on multiple reasoning benchmarks.
arXiv:2410. 12341v4 Announce Type: replace-cross Abstract: As AI-generated content increasingly populates the web, generative AI models are at growing risk of being trained on their own outputs, a process known as AI autophagy.
arXiv:2606. 11761v1 Announce Type: new Abstract: Dynamic data pruning techniques aim to reduce computational cost while minimizing information loss by periodically selecting representative subsets of input data during model training.
The paper introduces SynPro, a synthetic data generation framework that augments limited organic text for large language model pretraining by applying rephrasing and reformatting operations. SynPro’s generators are optimized with reinforcement learning rewards for quality, faithfulness, and data influence, and are updated continuously as training plateaus. Experiments on 400M, 1.1B, and 2B models show that SynPro can unlock 3.4–5.2× the effective tokens of standard repetition, even outperforming a non‑data‑bound oracle at larger scales.
Invent-A-Dataset is a prompt‑based system that generates realistic, large‑scale datasets from a description, targeting the zero‑data regime where no initial data exists. The authors benchmarked it against five leading model APIs across eight task types and up to 20,000 samples, finding that it outperforms competitors with 17% higher quality and 19% more diverse samples. The diversity advantage grows with dataset size, leading to better downstream training performance and consistently higher rankings for fine‑tuned models.
arXiv:2512. 15011v3 Announce Type: replace Abstract: Artificial intelligence (AI) increasingly generates the very content used to train future AI systems.
arXiv:2604. 17267v2 Announce Type: replace Abstract: Large Language Models can generate synthetic survey responses at low cost, but their accuracy varies unpredictably across questions.
Training-free few-shot adaptation methods have gained significant attention recently in the context of Vision-language Models (VLMs). Yet, current benchmarks rely on strong assumptions about the statistics of the adaptation data, e.