The paper investigates whether gradient-based training can learn the necessary rank to store and compose associations in a matrix memory. By training on $K$ key‑value bindings that require rank at least $K$ for exact linear recovery, the authors show that the learned effective rank increases with $K$ and that recovery sharply transitions near $k=K$ when rank is capped. Experiments demonstrate high recovery rates (cosine similarity >0.9) across multiple seeds and show that the trained operator approximates the ideal cycle on the entity subspace, though performance declines with larger matrix dimensions when encoder width is fixed.
arXiv:2609. 12259v1 Announce Type: new Abstract: Matrix-valued memories make rank the natural budget of a learned representation: the number of independent directions a state spans bounds what it can bind, compose, and track.
The paper investigates whether the rank of matrix-valued latent representations in continuous chain‑of‑thought models influences task accuracy. Experiments on ProsQA and GSM8K‑Aug show that truncating the latent matrix to low rank has negligible effect (within 0.6 pp), and this flatness persists across various readout designs and even in a vanilla GPT‑2 baseline. The results suggest that rank is not a useful structural signal for these models’ reasoning paths.
The paper investigates whether the rank of latent matrices in matrix‑chain‑of‑thought (Matrix‑CODI) models influences performance on reasoning tasks. Across multiple training regimes on ProsQA and GSM8K‑Aug, rank‑k projection ablations show flat accuracy curves, indicating that truncating the latent matrix to low rank does not hurt performance. Experiments with various readout architectures—bilinear, bilinear‑plus‑GELU, SVD‑augmented, and quadratic—confirm that rank‑indifference persists even for nonlinear readouts, and a linear probe on the latent matrix underperforms a raw pretrained hidden state.
arXiv:2606. 30067v1 Announce Type: cross Abstract: We introduce Neural Subspace Reallocation (NSR), which reframes continual learning as memory management over parameter subspaces.
arXiv:2605. 05189v2 Announce Type: replace-cross Abstract: How many key-value associations can a $d\times d$ linear memory store?
We introduce Neural Subspace Reallocation (NSR), which reframes continual learning as memory management over parameter subspaces. Instead of treating Low-Rank Adaptation (LoRA) modules as disposable per-task adapters, NSR manages them as compressible, retrievable memory units on a frozen backbone through a recurring cycle: (1) compress learned LoRAs via SVD, (2) reserve them in a TaskKnowledgeBank, (3) recall related past LoRAs by embedding similarity to warm-start new or returning tasks, and (4) reallocate the active subspace accordingly, with distillation protecting prior tasks.
arXiv:2607. 12204v1 Announce Type: new Abstract: Attention can be viewed as an online learner over context, yet existing test-time memories cannot certify that dropping a token leaves outputs unchanged or delete its influence outright.
arXiv:2606. 31813v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm.
arXiv:2608.22767v1 Announce Type: new Abstract: Long-term language-model agents accumulate memories across interactions, but their retrievers typically do not accumulate retrieval experience. Semanti...
arXiv:2610.23343v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) has become the standard method for parameter-efficient fine-tuning of large pretrained models, yet theoretical explanations...
The paper investigates how spectral statistics, particularly RankMe and covariance effective rank, behave when large language models (LLMs) undergo post‑training in non‑stationary environments. In controlled experiments on Qwen3‑0.6B, data duplication causes a 75% increase in held‑out loss while simultaneously raising both original and centred RankMe, indicating spectral dispersion rather than rank collapse. The authors demonstrate that a two‑sided, multichannel sequential monitor can detect various damage regimes but does not predict held‑out loss earlier and requires healthy calibration data.
arXiv:2609.16183v1 Announce Type: new Abstract: Fixed-state recurrences--linear attention and state-space models--are reported to lag behind attention on associative recall, but whole-architecture co...