arXiv AI
Aug 7
Supervised Learning Has a Geometric Blind Spot
arXiv:2604. 21395v3 Announce Type: replace-cross Abstract: Ordinary supervised training minimises the task loss and then stops.
The Matching Principle: When Does a Training Penalty Cover Deployment Shift?
arXiv:2605. 22800v3 Announce Type: replace-cross Abstract: Ordinary training optimises the task loss and then stops.
Related stories
arXiv Machine Learning
Jul 1
Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning
arXiv:2606. 31092v1 Announce Type: new Abstract: Full fine-tuning adapts large language models to new tasks but can erode capabilities they already possess.
arXiv Machine Learning
Jun 24
DREG: A Layer-Wise Jacobian Regularization as a General-Purpose Penalty
arXiv:2606. 23942v1 Announce Type: new Abstract: We present a large-scale empirical study isolating the contributions of the Derivative Regularization penalty (DREG).
arXiv Machine Learning
Jun 9
Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates
arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.
arXiv AI
Jul 22
Soft-TransFormers for Continual Learning
arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.
arXiv AI
1d ago
Exact Attention Sensitivity and the Geometry of Transformer Stability
arXiv:2602. 18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation.
arXiv Machine Learning
Jul 16
Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networks
arXiv:2607. 13380v1 Announce Type: new Abstract: Predictive Coding (PC) offers a biologically motivated alternative to backpropagation via local weight updates, yet routing error between layers still relies on an autograd Jacobian-transpose ($J^\top$) product - the last non-local operation in PC.
arXiv Machine Learning
Aug 4
Learned, Relied Upon, or Necessary? Separating Checkpoint Dependence from Task-Level Value in Sheaf GNNs
arXiv:2607. 25387v2 Announce Type: replace Abstract: Learned restriction maps in sheaf graph neural networks are often treated as proof that the model has discovered useful edge geometry.
arXiv AI
Jul 21
First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers
arXiv:2607. 16821v1 Announce Type: cross Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space.
arXiv Machine Learning
Aug 6
The Loss Does Not See the Basis, but Adam Does
arXiv:2608. 05136v1 Announce Type: new Abstract: Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not.
Hugging Face Trending Papers
Jun 28
Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks
A deep network's loss is invariant to continuous symmetries of its parameters: the logit shift, the ReLU rescaling, the LayerNorm scale, the per-head attention rotation. Adam's per-coordinate preconditioner drifts along each symmetry orbit, which pulls the trajectory off the symmetry quotient where the optimization lives and blurs the singular-learning rate the quotient makes readable.
arXiv Machine Learning
Aug 12
Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So
arXiv:2608. 10251v1 Announce Type: cross Abstract: A transformer's answer lives on one axis: the direction its unembedding reads.