Beyond LoRA: Can you beat the most popular fine-tuning technique?

Using LoRA for Efficient Stable Diffusion Fine-Tuning

(LoRA) Fine-Tuning FLUX.1-dev on Consumer Hardware

A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents

arXiv:2607. 17205v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) of open-weight LLMs on expert agent trajectories has emerged as a prominent approach to building capable code agents without reliance on proprietary models.

Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures

arXiv:2607. 21612v1 Announce Type: cross Abstract: Parameter-efficient fine-tuning methods like LoRA have become the default for adapting large language models, succeeding across instruction following, style transfer, and factual adaptation.

PLoRA: Efficient Concurrent LoRA Training for Large Language Models

arXiv:2508. 02932v2 Announce Type: replace Abstract: Low-Rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance.

Make LLM Fine-tuning 2x faster with Unsloth and 🤗 TRL

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

The paper shows that pretrained transformers often stop using their depth early, following only a few lines of context. A small rank‑8 LoRA applied to an early layer can extend this chain‑following ability, enabling models like Qwen3‑8B to achieve near‑perfect accuracy on 24‑line chains and significantly longer chains with further training. The LoRA acts as a relay, passing chain identity through middle layers and allowing frozen heads to read further up the chain, with the last useful intervention layer identified in most held‑out models.

\k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating

arXiv:2607. 22489v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become a widely adopted technique for efficient neural network fine-tuning, decomposing model updates into low-rank matrices.

ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

arXiv:2609.06072v1 Announce Type: cross Abstract: Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-w...

Parameter-Efficient Fine-Tuning using 🤗 PEFT

Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models

arXiv:2608. 07890v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert.