arXiv AI
Aug 7
Supervised Learning Has a Geometric Blind Spot
arXiv:2604. 21395v3 Announce Type: replace-cross Abstract: Ordinary supervised training minimises the task loss and then stops.
The Matching Principle: When Does a Training Penalty Cover Deployment Shift?
arXiv:2605. 22800v3 Announce Type: replace-cross Abstract: Ordinary training optimises the task loss and then stops.
Related stories
arXiv Machine Learning
1d ago
Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuning
arXiv:2610. 00436v1 Announce Type: new Abstract: Online batch selection fine-tunes a language model on the most useful part of each candidate batch.
arXiv Machine Learning
Jul 1
Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning
arXiv:2606. 31092v1 Announce Type: new Abstract: Full fine-tuning adapts large language models to new tasks but can erode capabilities they already possess.
arXiv Machine Learning
Jun 24
DREG: A Layer-Wise Jacobian Regularization as a General-Purpose Penalty
arXiv:2606. 23942v1 Announce Type: new Abstract: We present a large-scale empirical study isolating the contributions of the Derivative Regularization penalty (DREG).
arXiv Machine Learning
Jun 9
Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates
arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.
arXiv AI
Jul 22
Soft-TransFormers for Continual Learning
arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.
arXiv AI
Aug 18
Exact Attention Sensitivity and the Geometry of Transformer Stability
arXiv:2602. 18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation.
arXiv Machine Learning
Jul 16
Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networks
arXiv:2607. 13380v1 Announce Type: new Abstract: Predictive Coding (PC) offers a biologically motivated alternative to backpropagation via local weight updates, yet routing error between layers still relies on an autograd Jacobian-transpose ($J^\top$) product - the last non-local operation in PC.
arXiv Machine Learning
Sep 16
Noise2Noise Revisited: Training Pair Distributions Dominate Loss Choice in Self-Supervised Denoising
arXiv:2609.16788v1 Announce Type: new Abstract: Noise2Noise (N2N) trains denoisers on pairs of independently corrupted observations, eliminating clean references. We stress-test two natural conjectur...
arXiv Computer Vision
Aug 31
Activation Boundary Matching: Task-Informed Initialization for Low-Rank Adaptation
The paper introduces Activation Boundary Matching for Low‑Rank Adaptation (ABM‑LoRA), a task‑informed initialization strategy that uses the signs of layer‑wise pre‑activations from a brief probe adapter as targets for a fresh adapter. By training with a margin‑based hinge objective on these activation boundaries, ABM‑LoRA captures useful adaptation directions that standard LoRA initializers miss, while requiring only a few forward passes. Experiments show that ABM‑LoRA outperforms or matches existing LoRA, SVD, and gradient‑based initializers across multiple models and benchmarks, including T5‑base/GLUE, ConvNeXt‑T, Swin‑T, Qwen2.5‑1.5B, and LLaMA2‑7B.
arXiv Machine Learning
Aug 4
Learned, Relied Upon, or Necessary? Separating Checkpoint Dependence from Task-Level Value in Sheaf GNNs
arXiv:2607. 25387v2 Announce Type: replace Abstract: Learned restriction maps in sheaf graph neural networks are often treated as proof that the model has discovered useful edge geometry.
arXiv AI
Jul 21
First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers
arXiv:2607. 16821v1 Announce Type: cross Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space.