arXiv AI
Aug 7
Supervised Learning Has a Geometric Blind Spot
arXiv:2604. 21395v3 Announce Type: replace-cross Abstract: Ordinary supervised training minimises the task loss and then stops.
Reducing the Adaptation Gap Through Reachable Fisher Geometry
Related stories
arXiv Machine Learning
Aug 11
The Cost of Adaptivity: Matching Lower Bounds Across Learning Problems
arXiv:2608. 08826v1 Announce Type: new Abstract: Adaptive procedures must work without nuisance information an oracle may use, such as a gradient scale or smoothness index, and robust procedures may have to answer queries whose coordinate and inspection time are chosen only after the data are seen.
arXiv Machine Learning
2d ago
Profile-aware diagnostics of discarded parameter-Hessian mass
The paper evaluates block‑diagonal curvature models by measuring the squared off‑block fraction of the parameter Hessian across different group sizes and curvature profiles. Experiments on CIFAR‑100 ResNet‑18 and a skip‑free variant show that initial discarded fractions are similar but profiles differ, and that shifting from stage to layer groups changes the estimated off‑block mass relative to the profile envelope. Further comparisons on ResNet‑50/Stanford Cars examine off‑group Hessian actions, residuals, and combined candidates, revealing that off‑group input alone often increases minibatch loss, while combined candidates can reduce loss in some cases, with preferences depending on scale and correction‑norm matching.
arXiv Machine Learning
Aug 19
FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers
FishBack introduces a pullback Fisher geometry approach for activation steering in transformers, challenging the common Euclidean assumption of intermediate activation spaces. By deriving a closed‑form steering direction based on the Fisher information metric of the softmax layer, the method achieves target concept changes with minimal off‑target distortion, especially in early and middle layers. Experiments on GPT‑2 Small, Llama‑3‑8B, and Qwen3‑8B demonstrate significant reductions in off‑target KL divergence compared to existing steering baselines.
arXiv Machine Learning
Oct 7
How Inefficient Is Natural Gradient Descent? From Exact Optimality to \Theta ( \sqrt{ \log d } ) Divergence
arXiv:2610.07228v1 Announce Type: cross Abstract: Natural gradient descent (NGD) underlies common methods in ML. For dually flat families, idealized NGD on the forward Kullback--Leibler objective fol...
arXiv Machine Learning
Sep 18
FedFIbOS: Fisher Importance based Optimal Submodelling for Heterogeneous Federated Learning
FedFIbOS introduces a Fisher‑importance based criterion for selecting submodel parameters in heterogeneous federated learning, addressing the lack of theoretical justification in prior heuristic methods. By deriving a Fisher‑weighted quadratic masking surrogate and showing that the raw Fisher top‑k rule satisfies this surrogate under a Fisher‑dominant ranking condition, the method preserves convergence guarantees while efficiently estimating Fisher scores from squared gradients. Experiments on CIFAR‑10, CIFAR‑100, and AGNews demonstrate that FedFIbOS outperforms state‑of‑the‑art approaches by roughly 10% in accuracy, especially under strong non‑IID heterogeneity.
arXiv Machine Learning
Aug 4
When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer
arXiv:2608. 01130v1 Announce Type: new Abstract: A broad range of models face the mismatch where they are updated through trajectory losses but are evaluated by downstream task reward.
arXiv Machine Learning
Sep 2
Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation
arXiv:2609.00762v1 Announce Type: new Abstract: Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where...
arXiv Machine Learning
Aug 6
The Loss Does Not See the Basis, but Adam Does
arXiv:2608. 05136v1 Announce Type: new Abstract: Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not.
Hugging Face Trending Papers
Sep 1
Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation
Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential....
arXiv Machine Learning
Aug 4
Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models
arXiv:2608. 01624v1 Announce Type: cross Abstract: Adapting a language model to a task no longer requires training all of its weights, and a line of parameter-efficient methods has driven the trainable count from billions down to a handful of scalars.
arXiv Machine Learning
Jun 30
ITSPACE: Monotone Gaussian Optimal Transport Updates
arXiv:2606. 30523v1 Announce Type: new Abstract: Covariance matrices serve as compact descriptors of feature distributions in many machine-learning pipelines, including domain adaptation and Gaussian embeddings.