arXiv Computation and Language August 27, 2026 By Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike, Naoaki Okazaki

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Read the original on arXiv Computation and Language →

The paper introduces CoDIT, a contrastive decoding technique that separates instruction-following behavior from pre-trained world knowledge in large language models. By generating responses that emphasize post-training instruction capabilities while suppressing shared pre-trained knowledge, CoDIT creates instruction-tuning datasets that lead to consistently better model performance than directly generated responses or existing public datasets. The authors also provide theoretical and empirical evidence that CoDIT effectively distills instruction-tuning knowledge from model parameters into text, facilitating cross-architecture transfer.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Jul 14

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.

By Ajay Patel, Colin Raffel, Chris Callison-Burch

llms benchmarks

arXiv Machine Learning

Jul 10

Prompt Compression via Activation Aggregation

arXiv:2607. 08399v1 Announce Type: cross Abstract: Large language models process prompts by propagating activations through dozens of layers before generating a response.

By Thibaud Ardoin, Semira Einsele, Evis Bregu, Gerhard Wunder

llms

arXiv Machine Learning

Jun 19

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

arXiv:2602. 14696v2 Announce Type: replace Abstract: Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task.

By Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund, Sara Beery, David Alvarez-Melis

llms fine-tuning

arXiv AI

Jun 11

Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation

arXiv:2606. 12117v1 Announce Type: cross Abstract: Benchmark scores often misrepresent a large language model's (LLM's) knowledge, because they rely, e.

By Selen Erkan, Bastian Boll, Kristian Kersting, Bj\"orn Deiseroth, Letitia Parcalabescu

llms benchmarks

arXiv AI

Jun 9

InA-Probe: Instruction-Aware Active Probing for Time Series Forecasting with LLMs

arXiv:2606. 08601v1 Announce Type: new Abstract: Large Language Models (LLMs) have recently demonstrated impressive potential for time series forecasting.

By Peiliang Gong, Emadeldeen Eldele, Chenyu Liu, Ziyu Jia, Yi Ding, Xinliang Zhou, Lianchao Gu, Qi Zhu, Yang Liu, Daoqiang Zhang, Xiaoli Li

llms benchmarks safety

arXiv Computation and Language

3d ago

LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data

LuxIT is a monolingual instruction‑tuning dataset for Luxembourgish, created by synthesizing instruction‑answer pairs from native texts using the DeepSeek‑R1‑0528 model and a quality‑assurance LLM‑as‑judge process. The resulting 227,507 high‑quality pairs were used to fine‑tune 14 LLMs (≤15 B parameters), yielding an average accuracy increase of +5.37 percentage points on standardized Luxembourgish proficiency exams and improvements in macro‑averaged F1 on nine of the fourteen downstream NLP tasks. These findings demonstrate that synthetic monolingual data can effectively enhance LLM performance in low‑resource languages and reveal the complex relationship between exam performance and practical NLP gains.

By Julian Valline, Cedric Lothritz, Siwen Guo, Jordi Cabot

llms nlp fine-tuning benchmarks

One email a morning, machine-written

Related stories

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Prompt Compression via Activation Aggregation

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation

InA-Probe: Instruction-Aware Active Probing for Time Series Forecasting with LLMs

LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data