No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

arXiv:2607. 05872v1 Announce Type: new Abstract: Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly drifting object that can be tracked.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

• llms
• computer-vision