arXiv Computation and Language September 23, 2026 By Ashfak Yeafi, Mehedi Hasan, Md Khairul Islam

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning

Read the original on arXiv Computation and Language →

ChainDoRA is a new parameter‑efficient fine‑tuning framework for large language models that replaces the dense low‑rank factorization of LoRA with a connected Tensor‑Train (TT) chain. By separating weight magnitude and direction and using a TT rank to control representation capacity, ChainDoRA achieves higher average accuracy on seven commonsense reasoning benchmarks while dramatically reducing trainable parameters—down to 5.35 M versus 56 M for LoRA and DoRA. Ablation studies show that the TT parameterization offers controllable trade‑offs between parameter cost and accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

1d ago

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

arXiv:2609.37027v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-...

By Yihao Ouyang, Shiwei Li, Haozhao Wang, Xiandi Luo, Zhuoqi Hu, Jinglun Yu, Yichen Li, Ruixuan Li

computer-vision fine-tuning

arXiv Machine Learning

Sep 14

Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

The paper introduces ISO-LoRA, an optimizer that improves rank utilization in Low‑Rank Adaptation (LoRA) by coupling factor updates through spectral descent on the induced tangent perturbation in weight space. Experiments on GPT‑2 adaptation show that standard optimizers like AdamW concentrate updates in a few singular directions, whereas ISO-LoRA distributes energy more evenly, leading to higher effective rank and better downstream performance across 0.1B‑7B models. The authors provide theoretical guarantees under a stylized spiked‑gradient model and demonstrate that ISO-LoRA consistently outperforms factor‑wise optimizers, especially at moderate‑to‑large LoRA ranks.

By Zihan Zhu, Zhehang Du, Xuyang Chen, Tim Tsz-Kit Lau, Jiayuan Wu, X. Y. Han, Qi Long, Weijie Su

llms fine-tuning

arXiv Machine Learning

Jul 27

Layer-wise LoRA fine-tuning: a similarity metric approach

arXiv:2602. 05988v2 Announce Type: replace Abstract: Pre-training Large Language Models (LLMs) on web-scale datasets becomes fundamental for advancing general-purpose AI.

By Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Lucas Pellicer, Rosimeire Pereira Costa, Edson Bollis, Anna Helena Reali Costa, Artur Jordao

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Sep 23

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

arXiv:2609.25655v1 Announce Type: new Abstract: As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectu...

By Zhentao Tan, Chang Liu, Yao Liu, Yue Wu, Jieping Ye

llms fine-tuning benchmarks

Hugging Face Trending Papers

Sep 22

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectures such as Mixture-of-Experts (MoE) models. This...

llms fine-tuning benchmarks

arXiv AI

Jun 8

The Fine-Tuning Trap: Evaluating Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning

arXiv:2606. 06920v1 Announce Type: cross Abstract: Deploying Small Language Models (SLMs) on edge devices requires efficient fine-tuning strategies that adapt models to new tasks without degrading their general capabilities.

By Rahul Nair, Chun Tao

llms fine-tuning benchmarks

One email a morning, machine-written

Related stories

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

Layer-wise LoRA fine-tuning: a similarity metric approach

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

The Fine-Tuning Trap: Evaluating Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning