arXiv Machine Learning
From Parameter Rebasin to Functional Transfer Across Heterogeneous Transformers
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 31
Transporting Task Vectors across Different Architectures without Training
arXiv:2602. 12952v3 Announce Type: replace Abstract: Adapting large pre-trained models to downstream tasks often produces task-specific parameter updates that are expensive to relearn for every model variant.
arXiv Machine Learning
Jun 29
Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing
arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.
arXiv Machine Learning
Sep 15
Disentangling Representation Evolution in Transformers through Directional Decomposition
arXiv:2609.15975v1 Announce Type: cross Abstract: Transformer representations evolve through learned additive transformations that either preserve their current direction or redirect it. We study thi...
arXiv Machine Learning
Oct 15
Your Transformer Is Secretly Elastic
The paper presents a retraining‑free technique that transforms pre‑trained transformers into elastic models capable of adapting to any computational budget on the fly. By jointly ranking every MLP neuron and attention head on a common scale and applying soft pruning across multiple sparsity targets, the method achieves smooth performance degradation up to 60% sparsity while outperforming one‑shot baselines. Experiments on 27 vision and text transformer encoders show significant parameter reductions with minimal accuracy loss and notable throughput and latency gains on GPUs, all produced in minutes.
arXiv Machine Learning
Jun 4
Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform
arXiv:2603. 07523v3 Announce Type: replace Abstract: Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coupled with monolithic architecture, which restricts flexible reuse across models of varying scales.
arXiv Computer Vision
Sep 17
Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformers
Position Anchor Tuning (PAT) is a parameter‑efficient fine‑tuning method for pre‑trained point cloud transformers that improves inference efficiency. PAT reduces the computational cost of multi‑head attention and feed‑forward networks by using token aggregation‑expansion pairs: a token aggregation module (TAM) selects representative tokens based on 3D position anchors, and a token expansion module (TEM) propagates the learned representations back to the full token set. Combined with base‑sharing low‑rank adaptation (BSLoRA) for the TAMs, PAT achieves performance comparable to state‑of‑the‑art methods while requiring fewer trainable parameters and lower computational overhead.