arXiv Machine Learning August 4, 2026 By Taeyeong Kim, Ahhyun Kim, TaeHyeon Kim, Unggi Lee

Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models

Read the original on arXiv Machine Learning →

arXiv:2608. 01624v1 Announce Type: cross Abstract: Adapting a language model to a task no longer requires training all of its weights, and a line of parameter-efficient methods has driven the trainable count from billions down to a handful of scalars.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 21

First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

arXiv:2607. 16821v1 Announce Type: cross Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space.

By Irina Piontkovskaia, Sergey Nikolenko

llms fine-tuning

arXiv AI

1d ago

LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

arXiv:2602. 17510v2 Announce Type: replace-cross Abstract: We introduce LoRA-CRAFT (\textbf{C}ross-layer \textbf{R}ank \textbf{A}daptation via \textbf{F}rozen \textbf{T}ucker), abbreviated CRAFT throughout, an extremely parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors.

By Kasun Dewage, Marianna Pensky, Suranadi De Silva, Shankadeep Mondal

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 1

Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning

arXiv:2606. 31092v1 Announce Type: new Abstract: Full fine-tuning adapts large language models to new tasks but can erode capabilities they already possess.

By Rui Zhou, Tianci Xie

llms fine-tuning benchmarks

arXiv AI

Jul 1

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

arXiv:2606. 31813v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm.

By Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, Laixi Shi

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 21

Phantom Transitions in Language Model Fine-Tuning: A Density-Matrix Analysis

arXiv:2606. 07559v2 Announce Type: replace-cross Abstract: Fine-tuning a language model often fails silently when its correct completion must outrank a near-synonym competitor.

By Vaibhav Prakash, Jayasri Dontabhaktuni

llms rag fine-tuning

arXiv Machine Learning

Jun 9

Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates

arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.

By Edward Sun, Dmitrii Troitskii

fine-tuning benchmarks

Related stories

First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

Phantom Transitions in Language Model Fine-Tuning: A Density-Matrix Analysis

Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates