← Back to all news
arXiv Machine Learning By Borja Aizpurua, Sukhbinder Singh, Rom\'an Or\'us

Weight Clustering Reveals the Importance of Relative Rank in Large Language Models

Read the original on arXiv Machine Learning →

The study demonstrates that large language models can be compressed by clustering weight matrices into only 16–64 shared values without retraining, preserving strong accuracy. Fine‑tuning the cluster centroids recovers most of the remaining accuracy loss. Analysis reveals that maintaining the relative rank of weights and their overall variance is crucial for performance, while early layers and spatial geometry of cluster assignments also play significant roles.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 31

Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

The paper investigates the often-overlooked scale vectors in large language models, showing that despite their tiny size they are crucial for pre‑training performance. The authors provide theoretical insights that scale vectors mainly aid optimization rather than expressivity, and they analyze how weight decay affects different normalization layers. Building on these findings, they propose lightweight improvements—branch‑specific heterogeneity, better placement, and magnitude‑direction reparameterization—that consistently reduce loss across a range of model sizes and training settings.

By Mingze Wang, Shuchen Zhu, Yuxin Fang, Binghui Li, Kai Shen, Shu Zhong
llms
More like this →
arXiv Computation and Language
Sep 21

Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance

arXiv:2511.13254v2 Announce Type: replace Abstract: Large Language Models (LLMs) have displayed remarkable capabilities across diverse domains, but their training remains resource- and time-intensive...

By Shalini Maiti, Amar Budhiraja, Bhavul Gauri, Gaurav Chaurasia, Anton Protopopov, Alexis Audran-Reiss, Michael Slater, Despoina Magka, Tatiana Shavrina, Roberta Raileanu, Yoram Bachrach
llmsagentsbenchmarks
More like this →
arXiv Machine Learning
Sep 4

MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration

The paper introduces MSign, an optimizer designed to prevent training instability in large language models by restoring the stable rank of weight matrices. It identifies two precursors to gradient explosions—rapid stable rank decline and increased Jacobian alignment—and proves that these jointly cause exponential gradient growth. Experiments on models ranging from 5 M to 3 B parameters show that MSign stops training failures while adding less than 7.0% computational overhead.

By Lianhai Ren, Yucheng Ding, Xiao Liu, Peng Cheng, Yeyun Gong
llmssafety
More like this →
arXiv AI
Jul 21

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.

By Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu
llmsfine-tuningefficiencybenchmarks
More like this →