The paper investigates how synthetic data generated by large language models (LLMs) can be characterized using sample-level learnability derived from encoder training dynamics. It compares different LLM families and scales across tasks such as single- and multi-label classification, labeling, and tree prediction, and contrasts these synthetic datasets with human-written data. The study also examines the robustness of learned data distributions across encoders and evaluates how data selection strategies based on learnability signals impact the performance of both synthetic and organic data.
arXiv:2608. 11746v1 Announce Type: new Abstract: Modern systems are increasingly expected to transfer across tasks not specified during training.
The paper introduces FROST, an online framework that filters synthetic training data by estimating its utility through gradient feedback anchored in real data. FROST calibrates batch utility against recent history to decide when to filter, removing 20–30% of synthetic samples while improving performance on image classification and LLM fine-tuning tasks. The method is also applied to a large‑scale industrial ads re‑ranking system, yielding significant gains over an optimized production baseline.
arXiv:2509. 05130v2 Announce Type: replace Abstract: In classification problems, models are trained to predict a class label based on the input data features.
The paper introduces SynPro, a synthetic data generation framework that augments limited organic text for large language model pretraining by applying rephrasing and reformatting operations. SynPro’s generators are optimized with reinforcement learning rewards for quality, faithfulness, and data influence, and are updated continuously as training plateaus. Experiments on 400M, 1.1B, and 2B models show that SynPro can unlock 3.4–5.2× the effective tokens of standard repetition, even outperforming a non‑data‑bound oracle at larger scales.
arXiv:2606. 13629v1 Announce Type: cross Abstract: There is a proliferation of work arguing for the use of synthetic data in scientific research.
arXiv:2601. 17717v3 Announce Type: replace Abstract: Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities.
arXiv:2606. 00571v1 Announce Type: cross Abstract: Synthetic data are increasingly used to train neural networks, yet distributional mismatch with real data limits their effectiveness when used indiscriminately.
arXiv:2605. 09697v3 Announce Type: replace-cross Abstract: In many real-world computer vision applications, including medical imaging and industrial inspection, binary classification tasks are characterized by a severe scarcity of positive samples.
The paper proposes a three‑stage training pipeline that begins with procedural pretraining on abstract, procedurally generated data, followed by molecular pretraining on SMILES, and finally downstream fine‑tuning for molecular property prediction. Experiments show that procedural pretraining improves downstream performance—e.g., a 4.8% error reduction on Lipophilicity—especially when labeled data are scarce, and that the benefit peaks at an intermediate procedural training budget. Analysis indicates that transferable knowledge resides mainly in attention layers, while feed‑forward layers may over‑specialize.
arXiv:2607. 18072v1 Announce Type: cross Abstract: Generative models trained on a source domain often produce samples that are poorly aligned with shifted target domains, limiting their effectiveness for target-domain data augmentation.
arXiv:2602. 07298v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) represent a promising frontier for recommender systems, yet their development has been impeded by the absence of predictable scaling laws, which are crucial for guiding research and optimizing resource allocation.
The paper demonstrates that a tabular foundation model can achieve strong generalization using only a single real table for self‑supervised pre‑training, challenging the belief that large synthetic or real datasets are necessary. By systematically pre‑training and evaluating across diverse benchmarks, the authors show that the number and quality of tasks that can be derived from a dataset are critical for downstream performance. This finding suggests that carefully constructed task sets from limited data can enable effective transfer learning in tabular models.