arXiv AI
MoLS: SNR-Guided Module-Wise Learning-Rate Calibration for Large Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 10
One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
arXiv:2609.05885v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR...
arXiv Machine Learning
Jun 11
Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning
arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.
arXiv Machine Learning
Aug 18
SubZero+: Efficient Zeroth-Order LLM Fine-Tuning via Large Learning Rates
arXiv:2608. 15665v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables backpropagation-free fine-tuning of large language models, but existing ZO methods suffer from high-variance gradient estimators, making convergence unstable and highly sensitive to learning rates.
arXiv AI
Sep 1
LaMoC: Loss-Aware Modular Compression for LLMs
arXiv:2608.30226v1 Announce Type: new Abstract: Modular compression has enabled considerable parameter reduction in LLMs while preserving strong language understanding and downstream task accuracy. H...
arXiv Machine Learning
Sep 23
From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
arXiv:2609.25655v1 Announce Type: new Abstract: As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectu...
arXiv AI
Sep 24
Fine-Tune, Then Rectify
The paper proposes a two‑stage framework that first fine‑tunes a large language model (LLM) and then rectifies its outputs, allocating limited labeled data optimally between the stages. It argues that the usual mean‑squared‑error objective for fine‑tuning misaligns with the downstream rectification, and instead suggests minimizing prediction‑error variance for mean estimation or a scalarized variance metric for general M‑estimation. Empirical results confirm that this variance‑based fine‑tuning, combined with optimal data allocation, yields significant efficiency gains over using either fine‑tuning or rectification alone, or using the conventional objective.