The paper introduces SFT-as-context, a training‑free method that lets a parent large language model (LLM) use the response of a supervised fine‑tuned (SFT) model as context to answer queries. This approach preserves the parent model’s general capabilities while acquiring the SFT model’s specialized skills, achieving performance close to the SFT model on fine‑tuned benchmarks and near the parent model on general tasks. Experiments across 19 parent‑SFT pairs and 11 benchmarks show minimal gaps in performance, and the method can solve problems requiring both specialized and general knowledge, even when neither model alone succeeds.
arXiv:2606. 09525v1 Announce Type: cross Abstract: During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query.
The paper introduces CIG‑SFT, a supervised fine‑tuning method that uses conditional information gain to evaluate how much each demonstrated token reduces uncertainty about the task. By comparing token likelihoods from a frozen reference model with and without task input, the method derives a KL‑regularized projection of the current policy, yielding a scalar‑weighted SFT objective whose gradients match the projected target. Experiments on mathematical reasoning and code‑generation benchmarks show that CIG‑SFT outperforms standard SFT and adaptive fine‑tuning baselines, especially in generalizing to challenging evaluation distributions, without requiring extra data or online rollouts.
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
arXiv:2609.00759v1 Announce Type: new Abstract: Large language models (LLMs) increasingly handle in-context learning (ICL) tasks where a long, novel context defines the rules, knowledge, and output s...
The paper reports that chain‑of‑thought (CoT) supervised fine‑tuning (SFT) improves reasoning but systematically harms long‑context recall in hybrid linear‑attention models such as HypeNet and Jet‑Nemotron. Retrieval performance on the Needle‑In‑A‑Haystack benchmark drops dramatically after CoT‑SFT, especially with harder settings and longer contexts. The authors introduce QK‑Restore, a training‑free method that reinstates the query‑key projection matrices from the pre‑SFT checkpoint, which recovers long‑range recall while preserving reasoning gains.
Task-Level Natural Language Priors as Learning Signals for Low-Resource LLM Training proposes Prior-Guided Tuning (PGT), a training approach that treats natural-language priors as auxiliary learning signals rather than just input context. The method introduces Contrastive Prior Steering (CPS), which adds positive and negative prior-conditioned auxiliary losses while preserving the original supervised objective. Experiments on AmbiMath, Jigsaw, and MNLI/HANS demonstrate that CPS consistently outperforms plain and prompt fine-tuning, achieving high accuracy and significant gains with limited training data.
The paper introduces the concept of a prior barrier to measure how well pretrained large language models support different concepts during supervised fine‑tuning (SFT). It shows that these barriers follow a long‑tail distribution, causing head concepts to be easier to learn than tail concepts. To address this, the authors propose PASS, an adaptive instruction‑selection method that allocates limited supervision budget toward under‑covered concepts, outperforming seven state‑of‑the‑art methods across multiple settings.
arXiv:2607. 14111v1 Announce Type: cross Abstract: Can small language models detect and report on perturbations their own internal activations?
The paper introduces Predictability-Aware Canvas Selection and Supervision (PACS), a supervised fine‑tuning method for diffusion language models that aligns training with inference by using the model’s own predictability to determine which tokens are visible and when they are supervised. PACS selects the attention‑visible input length for each example and supervises reference tokens just before they become confident, addressing mismatches caused by random masking. Experiments on reasoning and code‑generation benchmarks show PACS consistently outperforms standard SFT, achieving up to 12.84 percentage‑point gains on individual tasks.
arXiv:2609.31688v2 Announce Type: replace Abstract: In verifiable domains such as math and coding, finding one correct solution among many attempts can matter more than the pass rate of each attempt....
arXiv:2607. 04733v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities.
arXiv:2607. 09415v1 Announce Type: cross Abstract: Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs.