arXiv Machine Learning
Reducing the Adaptation Gap Through Reachable Fisher Geometry
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 7
Supervised Learning Has a Geometric Blind Spot
arXiv:2604. 21395v3 Announce Type: replace-cross Abstract: Ordinary supervised training minimises the task loss and then stops.
arXiv Machine Learning
Aug 11
The Cost of Adaptivity: Matching Lower Bounds Across Learning Problems
arXiv:2608. 08826v1 Announce Type: new Abstract: Adaptive procedures must work without nuisance information an oracle may use, such as a gradient scale or smoothness index, and robust procedures may have to answer queries whose coordinate and inspection time are chosen only after the data are seen.
arXiv Machine Learning
2d ago
Profile-aware diagnostics of discarded parameter-Hessian mass
The paper evaluates block‑diagonal curvature models by measuring the squared off‑block fraction of the parameter Hessian across different group sizes and curvature profiles. Experiments on CIFAR‑100 ResNet‑18 and a skip‑free variant show that initial discarded fractions are similar but profiles differ, and that shifting from stage to layer groups changes the estimated off‑block mass relative to the profile envelope. Further comparisons on ResNet‑50/Stanford Cars examine off‑group Hessian actions, residuals, and combined candidates, revealing that off‑group input alone often increases minibatch loss, while combined candidates can reduce loss in some cases, with preferences depending on scale and correction‑norm matching.
arXiv Machine Learning
Aug 19
FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers
FishBack introduces a pullback Fisher geometry approach for activation steering in transformers, challenging the common Euclidean assumption of intermediate activation spaces. By deriving a closed‑form steering direction based on the Fisher information metric of the softmax layer, the method achieves target concept changes with minimal off‑target distortion, especially in early and middle layers. Experiments on GPT‑2 Small, Llama‑3‑8B, and Qwen3‑8B demonstrate significant reductions in off‑target KL divergence compared to existing steering baselines.
arXiv Machine Learning
Oct 7
How Inefficient Is Natural Gradient Descent? From Exact Optimality to \Theta ( \sqrt{ \log d } ) Divergence
arXiv:2610.07228v1 Announce Type: cross Abstract: Natural gradient descent (NGD) underlies common methods in ML. For dually flat families, idealized NGD on the forward Kullback--Leibler objective fol...
arXiv Machine Learning
Sep 18
FedFIbOS: Fisher Importance based Optimal Submodelling for Heterogeneous Federated Learning
FedFIbOS introduces a Fisher‑importance based criterion for selecting submodel parameters in heterogeneous federated learning, addressing the lack of theoretical justification in prior heuristic methods. By deriving a Fisher‑weighted quadratic masking surrogate and showing that the raw Fisher top‑k rule satisfies this surrogate under a Fisher‑dominant ranking condition, the method preserves convergence guarantees while efficiently estimating Fisher scores from squared gradients. Experiments on CIFAR‑10, CIFAR‑100, and AGNews demonstrate that FedFIbOS outperforms state‑of‑the‑art approaches by roughly 10% in accuracy, especially under strong non‑IID heterogeneity.