Parameter-Efficient Fine-Tuning using 🤗 PEFT

EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning

arXiv:2607. 01789v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation.

Approximate Muon with low-rank adapters

arXiv:2608. 14492v1 Announce Type: new Abstract: The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks.

Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

arXiv:2606. 14970v1 Announce Type: new Abstract: Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data.

GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models

arXiv:2506. 11042v2 Announce Type: replace Abstract: Parameter-efficient fine-tuning (PEFT) has emerged as a resource-efficient strategy for adapting Pretrained Foundation Models (PFMs) by learning a small number of task-specific updates $\Delta W$.

Make LLM Fine-tuning 2x faster with Unsloth and 🤗 TRL

The Fine-Tuning Trap: Evaluating Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning

arXiv:2606. 06920v1 Announce Type: cross Abstract: Deploying Small Language Models (SLMs) on edge devices requires efficient fine-tuning strategies that adapt models to new tasks without degrading their general capabilities.

Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps

20x Faster TRL Fine-tuning with RapidFire AI

Accelerating PyTorch distributed fine-tuning with Intel technologies

Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning

arXiv:2607. 18130v1 Announce Type: new Abstract: Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections.

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

arXiv:2609.25655v1 Announce Type: new Abstract: As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectu...

Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters

arXiv:2606. 00428v1 Announce Type: cross Abstract: Low-rank adapters are usually compared by sweeping a small set of ranks, but the rank also fixes the resolution of the parameter budget.