arXiv Machine Learning July 8, 2026 By Noel Thomas

No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

Read the original on arXiv Machine Learning →

arXiv:2607. 05872v1 Announce Type: new Abstract: Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly drifting object that can be tracked.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

4d ago

Scaling Zero-Order Pretraining through Model Sharding

arXiv:2609.37899v1 Announce Type: new Abstract: Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient...

By Francois Chaubard, Mykel J. Kochenderfer, Chris R\'e

arXiv Machine Learning

Aug 12

Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So

arXiv:2608. 10251v1 Announce Type: cross Abstract: A transformer's answer lives on one axis: the direction its unembedding reads.

By Mark Oskin

llms benchmarks

arXiv AI

Sep 1

Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization

arXiv:2608.29924v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) are prone to hallucinations: they fluently describe objects, attributes, and scenes that are not in the image. W...

By Aditi Sarker, Rafi Ibn Sultan, Hui Zhu, Dongxiao Zhu, Prashant Khanduri

llms rag fine-tuning multimodal safety

arXiv Machine Learning

Sep 23

Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention

arXiv:2608.28150v2 Announce Type: replace Abstract: How much matrix rank is required to preserve every bounded value output of normalized softmax attention? We study the unrestricted maximum-row-\(\e...

By Yuhe Sui, Jianing Zhang, Yingzhi Tang

llms

arXiv Machine Learning

Aug 6

The Loss Does Not See the Basis, but Adam Does

arXiv:2608. 05136v1 Announce Type: new Abstract: Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not.

By Devender Singh

llms safety

arXiv Machine Learning

Sep 11

EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale

EGGROLL replaces dense Gaussian perturbations in evolution strategies with low‑rank Gaussian products, enabling practical optimization of large language models while maintaining exactness on quadratic objectives. The paper analyzes the mean update field, error bounds, and shows that rank‑one perturbations add only a small variance penalty compared to dense ES. A new leave‑one‑out estimator, LOO‑ROLL, further reduces estimator MSE and improves post‑training performance on transformer blocks and GSM8K accuracy.

By Ege C. Kaya, Abolfazl Hashemi

llms

One email a morning, machine-written

Related stories

Scaling Zero-Order Pretraining through Model Sharding

Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So

Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization

Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention

The Loss Does Not See the Basis, but Adam Does

EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale