arXiv:2607. 24717v1 Announce Type: cross Abstract: Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs).
arXiv:2606. 07001v1 Announce Type: cross Abstract: High-quality training data is essential to large language models (LLMs) and typically requires extensive and costly manual curation.
arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.
arXiv:2607. 08646v1 Announce Type: cross Abstract: As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish.
arXiv:2608.29966v1 Announce Type: new Abstract: Domain adaptation of large language models increasingly depends on constructing high-quality training data, yet existing data-preparation pipelines typ...
arXiv:2607. 07748v1 Announce Type: new Abstract: Large Language Models achieve strong code generation for high resource languages like Python and Java but suffer sharp performance drops on Low-Resource Programming Languages~(LRPLs) such as Julia.
arXiv:2510. 01427v3 Announce Type: replace Abstract: At the core of Deep Research is knowledge mining, the task of extracting structured information from massive unstructured text in response to user instructions.
The paper explores ways to reduce the memorization of training data in language models, testing three regularizer-based, three finetuning-based, and eleven machine unlearning methods—five of which are newly introduced. It introduces TinyMem, a lightweight suite of small models for rapid testing of these mitigation techniques, and shows that unlearning methods, particularly BalancedSubnet, outperform others in removing memorized content while maintaining task performance. The study also finds that regularizer-based approaches are slow and ineffective, while finetuning methods are costly, especially when high accuracy is required.
RePro is a web‑recycling technique that trains a small language model (as little as 1 B parameters) with reinforcement learning to produce high‑quality, faithful rephrasings of pretraining data. The method uses one quality reward and three faithfulness rewards to preserve core semantics and structure while converting organic data into better training examples. Experiments show that RePro boosts downstream accuracy by 3.7–14.5 % over organic‑only baselines and improves data efficiency 2–3×, outperforming prior prompting‑based recycling approaches.
arXiv:2608. 20314v1 Announce Type: new Abstract: Mid-training is increasingly recognized as a critical stage for shaping the capabilities of large language models.
arXiv:2607. 16617v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to automate data-processing workflows, yet coding agents typically produce scripts that are not automatically materialized as persistent, editable platform artifacts.
arXiv:2604. 26258v3 Announce Type: replace-cross Abstract: LLM workflows, which coordinate structured calls to individual LLMs/agents to achieve a particular goal, offer a promising path towards building powerful AI systems that can tackle diverse tasks.
arXiv:2603. 03589v3 Announce Type: replace-cross Abstract: Recent advances in large language models (LLMs) transform how machine learning (ML) pipelines are developed and evaluated.