arXiv:2604. 13977v2 Announce Type: replace-cross Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent.
arXiv:2608. 25826v1 Announce Type: cross Abstract: A recent line of synthetic-data work reconstructs the thinking behind existing text rather than rewriting the text itself, but it operates on short web passages, recovers only local thoughts, and leaves the structure of whole documents untouched.
arXiv:2609.26347v1 Announce Type: cross Abstract: The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP)...
arXiv:2606.02991v2 Announce Type: replace-cross Abstract: While modern language models increasingly rely on ever-larger web corpora, we show that pretraining on historical text (e.g., pre-1913 text)...
arXiv:2609.00082v1 Announce Type: cross Abstract: LLMs acquire vast amounts of knowledge during pre-training, but often lack the specialized knowledge needed to answer questions from niche sources su...
AFCS (Atomic‑Fact‑Conditioned Synthesis) is a new pipeline for generating synthetic data for language‑model pretraining that focuses on extracting and using fine‑grained atomic facts from source documents. Unlike previous methods such as WRAP and Nemotron‑CC, AFCS explicitly requires secondary entities, list/table entries, and precise temporal attributes to appear in the synthesized corpus, employing structure‑aware extraction, source‑grounded verification, local context expansion, and multi‑format generation. In experiments with OLMo‑3‑7B and a 32B model on the FineWiki corpus, AFCS outperforms the baselines on several QA benchmarks and improves general benchmark performance by +1.09 when integrated mid‑training with 100B tokens.
A recent line of synthetic-data work reconstructs the thinking behind existing text rather than rewriting the text itself, but it operates on short web passages, recovers only local thoughts, and leav...
arXiv:2512.23065v4 Announce Type: replace Abstract: The introduction of BERT established encoder-only transformer models as a foundational paradigm in natural language processing. Encoder-only models...
arXiv:2606. 02991v1 Announce Type: cross Abstract: We introduce TypewriterLM, a 7.
LuxIT is a monolingual instruction‑tuning dataset for Luxembourgish, created by synthesizing instruction‑answer pairs from native texts using the DeepSeek‑R1‑0528 model and a quality‑assurance LLM‑as‑judge process. The resulting 227,507 high‑quality pairs were used to fine‑tune 14 LLMs (≤15 B parameters), yielding an average accuracy increase of +5.37 percentage points on standardized Luxembourgish proficiency exams and improvements in macro‑averaged F1 on nine of the fourteen downstream NLP tasks. These findings demonstrate that synthetic monolingual data can effectively enhance LLM performance in low‑resource languages and reveal the complex relationship between exam performance and practical NLP gains.
The paper presents a curated German–English parallel corpus derived from OPUS Books, reduced from 45,000 raw sentence pairs to 25,000 high‑quality examples through filtering and quality scoring. It details a reproducible fine‑tuning process for Meta’s NLLB‑200 models using mixed precision and gradient techniques, yielding a fine‑tuned NLLB‑600M model that improves BLEU, chrF++, and TER scores on literary and technical translations. The authors also integrate the model into the LinguaHaru translation system, enabling format‑preserving translation of PDFs and Office documents, and release all scripts, code, and the model on HuggingFace.
RT‑SFT is a method for text style transfer that uses roundtrip translation through a pivot language to strip stylistic information from monolingual corpora, creating pseudo‑parallel data. This data is then used to LoRA‑finetune an instruction‑tuned large language model as a stylizer, allowing the model to rewrite sentences in a target style while preserving meaning. Experiments across four style domains show that RT‑SFT surpasses state‑of‑the‑art approaches, including few‑shot in‑context learning, and offers effective retrieval augmentation for expert style domains with strict terminology.