The paper "Data Optimization for LLMs: A Survey" reviews data-related techniques for large language models, noting that while many surveys cover learning strategies and applications, few address data optimization. It outlines a framework that first discusses training and inference data challenges, then introduces dimensions for evaluating data properties—quality, difficulty, diversity, and redundancy—before categorizing existing optimization methods into data selection, augmentation, reweighting, and other strategies. The authors also examine interconnections among these categories and highlight future research directions compared to conventional deep learning approaches.
arXiv:2603.26164v2 Announce Type: replace-cross Abstract: Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters...
arXiv:2607. 10803v1 Announce Type: cross Abstract: Understanding which parameters are influential in Large Language Models (LLMs) is central to improving their efficiency, reliability, and interpretability.
arXiv:2607. 20465v1 Announce Type: new Abstract: The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end.
arXiv:2607. 18081v1 Announce Type: cross Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across a range of Natural Language Processing (NLP) tasks, but their high computational and memory demands pose significant challenges for deployment on resource-constrained edge devices.
BoLT is a new benchmark designed to make black-box optimization research for large language model (LLM) tasks more accessible. It presents a suite of realistic LLM optimization problems—covering multi‑fidelity, multi‑objective, heteroscedastic noise, and high‑dimensional search spaces—grounded in thousands of real experiments and represented by lightweight surrogate models. The authors evaluate a wide range of Bayesian and other black‑box optimization methods on BoLT, finding that some BO techniques consistently outperform others and revealing gaps in current methods for LLM optimization.
arXiv:2510. 16882v4 Announce Type: replace-cross Abstract: Supervised fine-tuning (SFT) is a commonly used technique to adapt large language models (LLMs) to downstream tasks.
arXiv:2504.04342v2 Announce Type: replace Abstract: Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly gro...
arXiv:2604. 26170v2 Announce Type: replace Abstract: Adapting large language models (LLMs) to a targeted task efficiently and effectively remains a fundamental challenge.
arXiv:2607. 03551v1 Announce Type: new Abstract: Weight space learning aims to learn representations of neural network (NN) weights, enabling different downstream tasks.
arXiv:2607. 22577v1 Announce Type: new Abstract: Scaling large language models (LLMs) has driven their success, yet dense Transformers couple capacity and computation: every parameter is activated for every token, making training and inference costs grow linearly with model size-a critical bottleneck as models approach trillion-parameter regimes.
arXiv:2602. 14696v2 Announce Type: replace Abstract: Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task.
arXiv:2505. 20161v2 Announce Type: replace-cross Abstract: Effective generalization in language models depends critically on the diversity of their training data.