The paper introduces OLGA, an online label‑free gradient matching method for continual language model training. OLGA synthesizes examples whose gradients match the model’s own next‑token predictions, enabling efficient updates on small unlabeled data sets. It achieves significant accuracy gains on five text classification benchmarks, outperforms existing selection‑based baselines in both offline and online settings, and scales to large data by combining selection and distillation in a hybrid pipeline.
Large language models (LLMs) struggle to classify text into taxonomies with many semantically similar labels, as the distinctions are domain-specific and not captured by pre-training. To handle large...
arXiv:2609.01564v1 Announce Type: cross Abstract: Large language models (LLMs) struggle to classify text into taxonomies with many semantically similar labels, as the distinctions are domain-specific...
arXiv:2609.37076v1 Announce Type: new Abstract: Large language models trained on vast corpora inherently risk memorizing harmful content that may later re-emerge in their outputs. To mitigate this is...
arXiv:2607. 18358v1 Announce Type: cross Abstract: Document classification is a solved problem in the laboratory and an unsolved one in the enterprise.
arXiv:2603.26164v2 Announce Type: replace-cross Abstract: Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters...
arXiv:2510. 16882v4 Announce Type: replace-cross Abstract: Supervised fine-tuning (SFT) is a commonly used technique to adapt large language models (LLMs) to downstream tasks.
The paper presents practical training recipes for looped language models, showing that a compute‑efficient pipeline can reduce the training budget from 7.7 T tokens to 310 B tokens while maintaining strong reasoning performance. In controlled experiments, a 1.4 B LoopLM outperforms a parameter‑matched dense model on 12 benchmarks, achieving significant gains on GSM8K, MATH, and DROP, and approaches a 3.9 B dense model at matched inference compute. The authors also provide a minimal recipe to convert pretrained dense models into looped ones, demonstrating improvements on Qwen3‑1.7B‑Base across multiple data regimes.
arXiv:2606. 02920v1 Announce Type: new Abstract: Language-model unlearning updates a trained model to behave as if it had not seen selected training examples, while preserving utility and avoiding costly retraining.
arXiv:2610.16182v1 Announce Type: new Abstract: Flagship large language models (LLMs) offer rich general knowledge and strong reasoning capabilities, but their high inference cost and latency limit t...
arXiv:2607. 24845v1 Announce Type: cross Abstract: Large language models (LLMs) have been applied to sequential recommendation by formulating it as a natural language task.
arXiv:2610.03702v1 Announce Type: new Abstract: The choice of post-training data for large language models substantially affects downstream performance. Gradient-based data selection is a popular app...
arXiv:2603. 12658v2 Announce Type: replace-cross Abstract: Continual learning (CL) has emerged as a pivotal paradigm to enable large language models (LLMs) to dynamically adapt to evolving knowledge and sequential tasks while mitigating catastrophic forgetting, a critical limitation of the static pre-training paradigm inherent to modern LLMs.