arXiv AI
Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 7
SemiAdapt-Instruct: Extensible Instruction Tuning via Latent Domain-Specialised Adapters
arXiv:2608. 05161v1 Announce Type: cross Abstract: Instruction-tuned LLMs are deployed into environments where domains evolve, yet extending a fine-tuned model's capabilities without full retraining remains an unsolved practical challenge.
arXiv AI
6d ago
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
arXiv Machine Learning
Jun 19
A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
arXiv:2602. 14696v2 Announce Type: replace Abstract: Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task.
arXiv Machine Learning
Aug 27
A Layer-wise Analysis of Supervised Fine-Tuning
The paper investigates how supervised fine‑tuning (SFT) affects different layers of language models, finding that middle layers (20–80%) remain stable while the final layers are highly sensitive to changes. Using information‑theoretic, geometric, and optimization metrics across 1B‑32B models, the authors identify a depth‑dependent pattern and introduce Mid‑Block Efficient Tuning, which updates only the critical intermediate layers. Experiments show this method outperforms standard LoRA by up to 10.2 % on GSM8K (OLMo2‑7B) with lower parameter overhead, suggesting that alignment can be achieved with localized architectural changes.
arXiv AI
Jul 20
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.
arXiv AI
Sep 30
Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
arXiv:2609.37027v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-...