← Back to all news
arXiv Machine Learning By Jiayu Bai, Danchen Yu, Zhenyu Liao, TianQi Hou, Feng Zhou, Robert C. Qiu, Zenan Ling

Diving into Kronecker Adapters: Component Design Matters

Read the original on arXiv Machine Learning →

arXiv:2602. 01267v3 Announce Type: replace Abstract: Kronecker adapters have emerged as a promising approach for fine-tuning large-scale models, enabling high-rank updates through tunable component structures.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 3

Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression

The paper introduces a scalable Kronecker-based approximation that captures cross-layer interactions without storing the full Fisher matrix, making Hessian analysis feasible for billion-parameter language models. It identifies consistent vulnerability patterns, notably that value projection layers are the most sensitive and exhibit strong cross-layer correlations across various model families. Experiments on quantization, sparsification, inter-layer corruption, and fine-tuning show that the approximation correlates strongly with performance degradation and recovery, providing a practical tool for identifying fragile components and guiding compression and optimization strategies.

By Viacheslav Yusupov, Daria Cherniuk, Evgeny Frolov
llmsfine-tuningefficiency
More like this →
arXiv Machine Learning
Sep 15

MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training

MoARa introduces a module-aware rank allocation strategy and a block-wise magnitude-direction decomposition to improve low-rank gradient projection for large language model pre‑training. By profiling Transformer modules and tailoring projection ranks, it reduces the number of steps and wall‑clock time needed to reach target perplexity. Experiments on Llama, Qwen, and DeepSeek models show up to 41.7% fewer steps and 37.1% less training time with minimal memory overhead.

By Keunyoung Kim, Nojun Kwak
llms
More like this →