Parameter-Efficient Fine-Tuning using 🤗 PEFT

EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning

arXiv:2607. 01789v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation.

Approximate Muon with low-rank adapters

arXiv:2608. 14492v1 Announce Type: new Abstract: The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks.

Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

arXiv:2606. 14970v1 Announce Type: new Abstract: Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data.

GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models

arXiv:2506. 11042v2 Announce Type: replace Abstract: Parameter-efficient fine-tuning (PEFT) has emerged as a resource-efficient strategy for adapting Pretrained Foundation Models (PFMs) by learning a small number of task-specific updates $\Delta W$.

Make LLM Fine-tuning 2x faster with Unsloth and 🤗 TRL

The Fine-Tuning Trap: Evaluating Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning

arXiv:2606. 06920v1 Announce Type: cross Abstract: Deploying Small Language Models (SLMs) on edge devices requires efficient fine-tuning strategies that adapt models to new tasks without degrading their general capabilities.

20x Faster TRL Fine-tuning with RapidFire AI

Accelerating PyTorch distributed fine-tuning with Intel technologies

Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning

arXiv:2607. 18130v1 Announce Type: new Abstract: Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections.

Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters

arXiv:2606. 00428v1 Announce Type: cross Abstract: Low-rank adapters are usually compared by sweeping a small set of ranks, but the rank also fixes the resolution of the parameter budget.

SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning

arXiv:2608. 15360v1 Announce Type: cross Abstract: While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing.

NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning

arXiv:2510. 18940v2 Announce Type: replace-cross Abstract: Existing parameter-efficient fine-tuning (PEFT) methods primarily fall into two categories: addition-based and selective in-situ adaptation.