← Back to all news
arXiv AI By Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Longtao Huang, Xiaochun Cao, Qingming Huang

ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning

Read the original on arXiv AI →

The paper introduces ToPA, a parameter‑efficient fine‑tuning method that replaces standard Toeplitz matrices with block‑wise Toeplitz matrices to increase capacity and shorten multiplicative chains, thereby stabilizing training and enhancing expressiveness. The authors theoretically prove that products of block‑wise Toeplitz matrices can approximate arbitrary matrices with shorter chain lengths and empirically demonstrate that ToPA outperforms existing low‑rank and sparse parameterizations across 21 NLP and CV datasets and five model architectures.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 2

GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioning

GPart introduces a new parameter‑efficient fine‑tuning technique that directly maps a low‑dimensional trainable vector into the full weight space using a sparse, isometric partition matrix. Unlike LoRA, GPart eliminates the bilinear reconstruction step, preserving exact end‑to‑end isometry and reducing the checkpoint to just the vector and a random seed. Experiments across NLP, vision, and reasoning tasks show that GPart matches or surpasses existing PEFT methods while using far fewer parameters and offering a simpler, more tractable parameterization.

By Paolo Mandica, Micha{\l} Brzozowski, Zuzanna Dubanowska, Neo Christopher Chung
computer-visionfine-tuningbenchmarks
More like this →
arXiv AI
Jul 21

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.

By Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu
llmsfine-tuningefficiencybenchmarks
More like this →