arXiv:2606. 19491v1 Announce Type: new Abstract: Pretrained transformers sit near singular minima of the loss, where the Fisher information metric degenerates along dead directions: directions in parameter space along which the directional Fisher vanishes.
arXiv:2607. 00603v1 Announce Type: new Abstract: We give a descent-free, alignment-free measurement of singular structure on trained networks.
A deep network's loss is invariant to continuous symmetries of its parameters: the logit shift, the ReLU rescaling, the LayerNorm scale, the per-head attention rotation. Adam's per-coordinate preconditioner drifts along each symmetry orbit, which pulls the trajectory off the symmetry quotient where the optimization lives and blurs the singular-learning rate the quotient makes readable.
arXiv:2606. 29176v1 Announce Type: new Abstract: A deep network's loss is invariant to continuous symmetries of its parameters: the logit shift, the ReLU rescaling, the LayerNorm scale, the per-head attention rotation.
arXiv:2606. 05957v1 Announce Type: new Abstract: Singular learning theory and information geometry have studied the same parameter spaces in mostly separate vocabularies: the former computes Bayesian invariants in resolved coordinates, the latter works in original coordinates under a non-degeneracy assumption that overparameterised models routinely violate.
arXiv:2608. 10251v1 Announce Type: cross Abstract: A transformer's answer lives on one axis: the direction its unembedding reads.
arXiv:2606. 31963v1 Announce Type: new Abstract: Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-$k$ neuron sets, attribution lists, and merge alignments.
arXiv:2609.37717v1 Announce Type: new Abstract: Decoder-only transformers are trained only through a terminal next-token prediction loss, yet this loss constrains every intermediate hidden state thro...
arXiv:2607. 16821v1 Announce Type: cross Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space.
The paper introduces a reference‑free instrument that, from a single fit and without an oracle, can detect whether a hybrid PDE‑parameter estimator’s assumed operator is misspecified and distinguish this from mere parameter unidentifiability. In a self‑adjoint parabolic inverse problem, the proposed information‑matrix statistic correctly identifies misspecification with low false‑positive rates, while remaining silent when the design is correctly specified but non‑identifiable. The study demonstrates that conventional accuracy checks can miss significant operator errors, and it maps out the instrument’s blind spots and conditions under which its guarantees hold.
arXiv:2607. 23711v1 Announce Type: new Abstract: LoRA fine-tuning can create intruder dimensions: new leading singular vectors of the updated weight matrix $W+BA$ that are nearly orthogonal to all pretrained singular vectors and that drive catastrophic forgetting.
arXiv:2606. 02596v1 Announce Type: new Abstract: The curvature exponent $\alpha$ in $h_k \propto \sigma_k^\alpha$ -- governing how Hessian eigenvalues scale with gradient singular values -- varies systematically across layer types ($\alpha \approx 2$ for convolutions, $\approx 1$ for transformer attention, $< 1$ for MLP up-projections).
arXiv:2608. 25138v1 Announce Type: new Abstract: Stochastic masking, cropping, or modality removal makes deterministic reconstruction an incomplete target: one observation can admit many clean completions.