arXiv Machine Learning
Bilinear Optimization Divergence: Diagnosing Factor-Constrained LoRA Continual Learning
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 2
Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation
arXiv:2609.00762v1 Announce Type: new Abstract: Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where...
Hugging Face Trending Papers
Sep 1
Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation
Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential....
arXiv AI
Sep 15
LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference
LayerRoute is a parameter‑efficient technique that enables adaptive skipping of transformer layers in large language models. It adds a lightweight per‑layer router (~21.5K parameters) and LoRA adapters (rank 8, ~1.08M parameters) to each of the 24 blocks in Qwen2.5‑0.5B‑Instruct, training them jointly with a gate‑regularized language‑modeling objective. Across ten independent runs, the method consistently identifies nine middle layers as skip‑eligible, achieves a verified wall‑clock speedup of 1.02x–1.06x, and improves perplexity by an average of 1.16 points, while the router’s decisions vary per input, confirming genuine adaptive behavior.
arXiv Machine Learning
Sep 22
Muon Can Outperform Dedicated Continual Learning Methods
arXiv:2609.24678v1 Announce Type: new Abstract: Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past...
arXiv AI
Jul 21
First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers
arXiv:2607. 16821v1 Announce Type: cross Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space.
arXiv Machine Learning
Oct 15
Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning
The paper introduces LoDA, a LoRA-based continual learning method that decomposes update spaces into task‑shared and truly task‑specific subspaces using energy‑based objectives. By fixing down‑projections and learning up‑projections with Gradient‑Aligned Optimization, LoDA recalibrates general updates in closed form after each task, aiming to improve knowledge sharing and isolation. Experiments show that LoDA outperforms existing CL methods.