arXiv Computation and Language August 27, 2026 By Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike, Naoaki Okazaki

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Read the original on arXiv Computation and Language →

The paper introduces CoDIT, a contrastive decoding technique that separates instruction-following behavior from pre-trained world knowledge in large language models. By generating responses that emphasize post-training instruction capabilities while suppressing shared pre-trained knowledge, CoDIT creates instruction-tuning datasets that lead to consistently better model performance than directly generated responses or existing public datasets. The authors also provide theoretical and empirical evidence that CoDIT effectively distills instruction-tuning knowledge from model parameters into text, facilitating cross-architecture transfer.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Jul 14

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.

By Ajay Patel, Colin Raffel, Chris Callison-Burch

llms benchmarks

arXiv Machine Learning

Jul 10

Prompt Compression via Activation Aggregation

arXiv:2607. 08399v1 Announce Type: cross Abstract: Large language models process prompts by propagating activations through dozens of layers before generating a response.

By Thibaud Ardoin, Semira Einsele, Evis Bregu, Gerhard Wunder

llms

arXiv Machine Learning

Jun 19

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

arXiv:2602. 14696v2 Announce Type: replace Abstract: Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task.

By Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund, Sara Beery, David Alvarez-Melis

llms fine-tuning

arXiv AI

Sep 3

DKL: Decoupled Knowledge Learning for Instruction-Tuned Language Models

The paper introduces DKL, a method for adding new knowledge to instruction‑tuned language models without compromising their instruction‑following abilities. DKL performs extended pre‑training on a base LLM to embed knowledge, then merges these weights into the instruction‑tuned model, avoiding costly instruction fine‑tuning. Experiments show DKL raises RAG accuracy from 54.17% to 79.26% on retrieval failure cases while using far less training data than previous approaches.

By Kushagra Bhushan, Meghanadh Pulivarthi, Sai Krishna Reddy Sathi, Gaurav Pandey, Sonam Gupta, Vineet Kumar, Jaydeep Sen, Yatin Nandwani, Sachindra Joshi, Dinesh Raghu

llms rag fine-tuning safety

arXiv AI

Sep 12

How LLMs Follow Instructions: Skillful Coordination, Not a Universal Mechanism

Instruction tuning is often thought to give language models a universal ability to follow instructions, but this study shows otherwise. By probing nine tasks across three models, the authors find that general probes reveal selective, not uniform, deficits, cross‑task transfer is weak and skill‑similar, and causal ablation uncovers sparse, asymmetric dependencies. The results suggest instruction following is a coordinated use of diverse linguistic skills rather than a single shared mechanism.

By Elisabetta Rocchetti, Alfio Ferrara

llms fine-tuning benchmarks

arXiv AI

Jun 11

Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation

arXiv:2606. 12117v1 Announce Type: cross Abstract: Benchmark scores often misrepresent a large language model's (LLM's) knowledge, because they rely, e.

By Selen Erkan, Bastian Boll, Kristian Kersting, Bj\"orn Deiseroth, Letitia Parcalabescu

llms benchmarks

One email a morning, machine-written

Related stories

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Prompt Compression via Activation Aggregation

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

DKL: Decoupled Knowledge Learning for Instruction-Tuned Language Models

How LLMs Follow Instructions: Skillful Coordination, Not a Universal Mechanism

Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation