Matricized low-rank approximation via SVD is a standard surrogate for tensor decompositions, but entry-wise reconstruction error fails to capture multiway geometric degradation. Under an orthogonal Tucker model, we characterize this degradation using two metrics: cross-mode Direction Loss, measuring geometric subspace deviation from rank truncation and noise rotation, and Interaction Loss, quantifying multilinear interaction distortion in the core tensor.
arXiv:2609.28029v1 Announce Type: cross
Abstract: The key-value (KV) cache of autoregressive transformers can be viewed as a fourth-order tensor spanning attention heads, tokens, features, and groupe...
By Rahul Krishnan, Volker Schulz
The key-value (KV) cache of autoregressive transformers can be viewed as a fourth-order tensor spanning attention heads, tokens, features, and grouped layers. We measure the singular-value spectra of...
arXiv:2608. 05136v1 Announce Type: new Abstract: Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not.
By Devender Singh
The paper introduces MultiHU‑TD, an interpretable framework for multifeature hyperspectral unmixing that employs tensor decomposition and incorporates the abundance sum‑to‑one constraint via an ADMM algorithm. It extends previous models by adding mathematical morphology and neighborhood patch analysis, and provides detailed mathematical, physical, and graphical interpretations linked to the extended linear mixing model. Experiments on real hyperspectral images demonstrate the model’s interpretability and effectiveness, with code released on GitHub.
By Mohamad Jouni, Mauro Dalla Mura, Lucas Drumetz, Pierre Comon
arXiv:2609.14307v1 Announce Type: new
Abstract: Low-rank tensor factorization provides a flexible framework for completing multidimensional data from incomplete and corrupted observations. However, u...
By Binghao Wang, Feng Zhang, Wendong Wang, Jianjun Wang
arXiv:2610.08533v1 Announce Type: cross
Abstract: Geometric alignment scores based on Gram determinants provide a compact way to model higher-order consistency among modalities, yet how such scores r...
By Yongsheng Luo, Wengan He, Yu Li, Rouying Wu, Wei Lv
arXiv:2608.30374v1 Announce Type: cross
Abstract: We study null-space estimation from a noisy matrix. For a simple left null space, we first derive an exact compact expression for the error of the sm...
By Xin Li, Jonathan Cohen, Rami Puzis
arXiv:2510. 07758v3 Announce Type: replace Abstract: Sharpness (of the loss minima) is widely believed to be a good indicator of generalization of neural networks.
By Qiaozhe Zhang, Jun Sun, Ruijie Zhang, Yingzhuang Liu
arXiv:2606. 04405v1 Announce Type: cross Abstract: Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes.
By Mingyu Li
The paper investigates how pretraining and alignment stages of transformer training affect weight updates in different matrix groups. By examining the residual stream’s shared coordinate system, it finds that alignment updates to query and key matrices align with the principal input subspace, whereas updates to output matrices remain largely random relative to the prediction subspace. This pattern persists across objectives and training stages, indicating that the geometry of updates is determined by the training phase rather than the specific objective.
By Valeria Ruscio, Keiran Thompson
The paper introduces a new concept called positive scattering to enhance identifiability of nonnegative tensor decompositions. By combining this scattering term with existing dimension-based conditions, the authors derive two sufficient criteria that guarantee minimality, nonnegative rank, and uniqueness for subsets of components. The key result is a positive splitting inequality that links dimension constraints with support-induced geometric rigidity, and the authors show that the scattering term’s mode costs are discrete, enabling an exact activation characterization via graph connectivity. This criterion can certify sparse nonnegative tensor decompositions that elude traditional Kruskal and Lovitz–Petrov conditions, even after reshaping, and reduces to familiar matrix results in the two-dimensional case.