arXiv Computation and Language October 16, 2026 By Guangzeng Han, Xiaolei Huang

What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective

Read the original on arXiv Computation and Language →

The paper introduces a weighted in-context influence (wICI) framework for selecting instruction‑tuning data, aiming to reduce redundancy and improve sample quality. It investigates what makes data effective from an in‑context perspective, explores the relationship between sample difficulty and influence, and evaluates how influence predicts tuning performance. Experiments on various models and benchmarks show that wICI consistently outperforms existing baselines under limited data budgets and reveals a negative correlation between sample difficulty and in‑context influence.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language

Oct 27

"High-Quality" Data for Early-Stage Instruction Tuning: An Empirical Study of Data Quality Metrics

arXiv:2610.33969v1 Announce Type: new Abstract: "Data quality" is widely invoked in instruction tuning, but the term is often underspecified. We study it as an operational property by asking which si...

By Vladimir Necula, David Gvadzabia, Sofia Serrano

llms fine-tuning data-engineering

arXiv Computation and Language

Sep 16

In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

The paper investigates whether in‑context learning (ICL) can replace instruction tuning for multilingual language models, especially as model size varies. It highlights the difficulty of obtaining high‑quality instruction data in multilingual settings and compares the performance of ICL versus instruction‑tuned models. The findings show that a performance gap persists between the two approaches, suggesting the need for further research to close it.

By David Ponce, Thierry Etchegoyhen

llms fine-tuning safety

arXiv Machine Learning

Jun 19

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

arXiv:2602. 14696v2 Announce Type: replace Abstract: Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task.

By Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund, Sara Beery, David Alvarez-Melis

llms fine-tuning

arXiv AI

Jun 9

Emergence of Context Characteristics Sensitivity in Large Language Models

arXiv:2606. 09525v1 Announce Type: cross Abstract: During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query.

By Nadya Yuki Wangsajaya, Haeun Yu, Isabelle Augenstein

llms reinforcement-learning fine-tuning

arXiv AI

Jun 18

DRIFT: Refining Instruction Data via On-Policy Data Attribution

arXiv:2606. 18307v1 Announce Type: cross Abstract: Optimizing the training data distribution for Supervised Fine-Tuning (SFT) dictates the capability of Large Language Models (LLMs).

By Zefan Wang, Lincheng Li, Tianyu Yu, Yuan Yao

llms fine-tuning safety

arXiv AI

Aug 14

Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity

arXiv:2608. 13430v1 Announce Type: cross Abstract: Instruction-tuned language models achieve strong performance across a range of generation tasks, but have also recently been shown to exhibit verbalized overconfidence.

By Irina Proskurina, Mayank Kumar, Oyindolapo O. Komolafe

llms nlp fine-tuning benchmarks

One email a morning, machine-written

Related stories

"High-Quality" Data for Early-Stage Instruction Tuning: An Empirical Study of Data Quality Metrics

In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

Emergence of Context Characteristics Sensitivity in Large Language Models

DRIFT: Refining Instruction Data via On-Policy Data Attribution

Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity