The paper introduces XTF, an explainable token‑level noise filtering framework for fine‑tuning large language models. XTF breaks down token contributions into reasoning importance, knowledge novelty, and task relevance, scores them, and masks gradients of noisy tokens to improve fine‑tuning. Experiments on math, code, and medicine tasks across seven LLMs show up to a 13.7% performance boost over standard fine‑tuning.
arXiv:2604. 26170v2 Announce Type: replace Abstract: Adapting large language models (LLMs) to a targeted task efficiently and effectively remains a fundamental challenge.
The paper proposes an information-weighted cross‑entropy loss that rescales token contributions using TF‑IDF statistics, thereby emphasizing semantically informative tokens and down‑weighting ubiquitous ones. Experiments on five decoder‑only language models (1.1B–13B parameters) show consistent reductions in memorized substring length while maintaining perplexity and downstream performance. The method is architecture‑agnostic, adds less than 3% computational overhead, and can be integrated into existing training pipelines.
arXiv:2602. 11177v2 Announce Type: replace-cross Abstract: Reliable early detection of Alzheimer's disease (AD) is challenging, particularly due to the limited availability of labeled data.
DECSELFMASK is a decoder‑only classification method that uses unlabeled clinical text to improve performance. It creates self‑supervised training examples by masking portions of the text identified as relevant through relevance attribution, then trains the model to reconstruct the masked tokens via next‑token prediction. Experiments on 136 tasks from 1.9 M Italian hospital notes show consistent gains across five models, outperforming base models (+9.1 Macro F1), continual pretraining (+6.3), and synthetic label generation (+12.5).
Large language models are typically trained under uniform token weighting, which allows frequent and low-information tokens to dominate learning and can increase the tendency to memorize surface-level...
arXiv:2609.13141v1 Announce Type: new Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context u...
The paper introduces TokenAdapt, a model‑agnostic tokenizer transplantation method that uses a hybrid heuristic to initialize new token embeddings, and a novel pre‑tokenization learning approach for multi‑word Supertokens to improve compression. TokenAdapt combines local subword decomposition and global semantic similarity to preserve semantics while reducing retraining needs. Empirical results show that TokenAdapt outperforms existing baselines such as Transtokenizer and ReTok, achieving lower perplexity ratios and significant compression gains.
arXiv:2506.19549v2 Announce Type: replace-cross Abstract: Identifying which tokens and activations truly influence model predictions is critical for both the efficiency and interpretability of large...
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
The paper introduces CIG‑SFT, a supervised fine‑tuning method that uses conditional information gain to evaluate how much each demonstrated token reduces uncertainty about the task. By comparing token likelihoods from a frozen reference model with and without task input, the method derives a KL‑regularized projection of the current policy, yielding a scalar‑weighted SFT objective whose gradients match the projected target. Experiments on mathematical reasoning and code‑generation benchmarks show that CIG‑SFT outperforms standard SFT and adaptive fine‑tuning baselines, especially in generalizing to challenging evaluation distributions, without requiring extra data or online rollouts.
arXiv:2606. 24841v1 Announce Type: new Abstract: Prompt-based learning has emerged as a dominant paradigm in natural language processing.
arXiv:2606. 11189v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory.