← Back to all news
arXiv Machine Learning By Mingze Wang, Shuchen Zhu, Yuxin Fang, Binghui Li, Kai Shen, Shu Zhong

Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

Read the original on arXiv Machine Learning →

The paper investigates the often-overlooked scale vectors in large language models, showing that despite their tiny size they are crucial for pre‑training performance. The authors provide theoretical insights that scale vectors mainly aid optimization rather than expressivity, and they analyze how weight decay affects different normalization layers. Building on these findings, they propose lightweight improvements—branch‑specific heterogeneity, better placement, and magnitude‑direction reparameterization—that consistently reduce loss across a range of model sizes and training settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 12

Weight Clustering Reveals the Importance of Relative Rank in Large Language Models

The study demonstrates that large language models can be compressed by clustering weight matrices into only 16–64 shared values without retraining, preserving strong accuracy. Fine‑tuning the cluster centroids recovers most of the remaining accuracy loss. Analysis reveals that maintaining the relative rank of weights and their overall variance is crucial for performance, while early layers and spatial geometry of cluster assignments also play significant roles.

By Borja Aizpurua, Sukhbinder Singh, Rom\'an Or\'us
llmsfine-tuning
More like this →
arXiv Machine Learning
Aug 28

Squeezing More from Limited Data with Recursive Transformers

The paper investigates how to effectively pre‑train language models when the data budget is limited but compute is plentiful. It shows that increasing model size only improves performance up to an optimal point, after which overfitting degrades generalization, and that this optimal size varies with both the data budget and downstream tasks. To overcome the inefficiencies of standard Transformers in this regime, the authors propose recursive Transformers that reuse a shared block across depth and employ factorized embeddings, achieving better results than standard models on 10M–100M word pre‑training budgets and competitive performance with BabyLM Challenge 2025 winners.

By Serdar G\"ulbahar, Lukas Edman, Alexander Fraser
llmsrag
More like this →