The paper introduces a weighted in-context influence (wICI) framework for selecting instruction‑tuning data, aiming to reduce redundancy and improve sample quality. It investigates what makes data effective from an in‑context perspective, explores the relationship between sample difficulty and influence, and evaluates how influence predicts tuning performance. Experiments on various models and benchmarks show that wICI consistently outperforms existing baselines under limited data budgets and reveals a negative correlation between sample difficulty and in‑context influence.
arXiv:2610.33969v1 Announce Type: new Abstract: "Data quality" is widely invoked in instruction tuning, but the term is often underspecified. We study it as an operational property by asking which si...
The paper investigates whether in‑context learning (ICL) can replace instruction tuning for multilingual language models, especially as model size varies. It highlights the difficulty of obtaining high‑quality instruction data in multilingual settings and compares the performance of ICL versus instruction‑tuned models. The findings show that a performance gap persists between the two approaches, suggesting the need for further research to close it.
arXiv:2602. 14696v2 Announce Type: replace Abstract: Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task.
arXiv:2606. 09525v1 Announce Type: cross Abstract: During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query.
arXiv:2606. 18307v1 Announce Type: cross Abstract: Optimizing the training data distribution for Supervised Fine-Tuning (SFT) dictates the capability of Large Language Models (LLMs).
arXiv:2608. 13430v1 Announce Type: cross Abstract: Instruction-tuned language models achieve strong performance across a range of generation tasks, but have also recently been shown to exhibit verbalized overconfidence.
The paper introduces CoDIT, a contrastive decoding technique that separates instruction-following behavior from pre-trained world knowledge in large language models. By generating responses that emphasize post-training instruction capabilities while suppressing shared pre-trained knowledge, CoDIT creates instruction-tuning datasets that lead to consistently better model performance than directly generated responses or existing public datasets. The authors also provide theoretical and empirical evidence that CoDIT effectively distills instruction-tuning knowledge from model parameters into text, facilitating cross-architecture transfer.
arXiv:2608. 12218v1 Announce Type: cross Abstract: Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories.
The paper evaluates 17 paradigm-level configurations of in‑context learning (ICL) text‑to‑SQL pipelines across five common modules, measuring each module’s marginal accuracy contribution and cost for four different backbone models. It finds that execution‑feedback refinement consistently improves accuracy at low cost, while other modules only help under specific backbone conditions. The study also shows that investing in a more elaborate pipeline for a mid‑tier backbone can be more cost‑effective than upgrading to a high‑capability model with a lean pipeline, providing a tiered, cost‑aware guideline that generalizes to additional backbones.
arXiv:2501. 12147v2 Announce Type: replace-cross Abstract: Selecting appropriate training data is crucial for instruction fine-tuning of large language models (LLMs), which aims to (1) elicit strong capabilities, and (2) achieve balanced performance across different tasks.
arXiv:2503. 06573v3 Announce Type: replace-cross Abstract: Recent LLMs have shown remarkable success in following user instructions, yet handling instructions with multiple constraints remains a significant challenge.
LIMIT (Less Is More for Instruction Tuning in Text-to-SQL) challenges the belief that large instruction corpora are necessary for effective Text-to-SQL models. The framework uses a four‑stage data‑centric process—difficulty‑aware filtering, chain‑of‑thought synthesis, LLM‑as‑judge quality scoring, and genetic algorithm optimization—to select a compact set of examples that still achieve full schema coverage. On the BIRD and Spider benchmarks, LIMIT’s 796 and 863 samples enable Qwen3‑8B to reach 69.1% and 88.9% execution accuracy, outperforming methods trained on twenty times more data and setting a new state‑of‑the‑art for open‑source approaches.