arXiv Machine Learning
Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 31
InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
InfoMamba is an attention‑free hybrid model that combines a minimal‑bandwidth global interface with a selective recurrent stream. The architecture replaces token‑level self‑attention with a concept bottleneck linear filtering layer and integrates it via an information‑maximizing fusion (IMF) that injects global context into the state‑space dynamics. Experiments across classification, dense prediction, and non‑vision tasks show that InfoMamba outperforms strong Transformer and SSM baselines while maintaining near‑linear scaling and competitive accuracy‑efficiency trade‑offs.
arXiv AI
Jun 30
Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers
arXiv:2510. 25013v2 Announce Type: replace-cross Abstract: Mechanistic interpretability aims to reverse-engineer large language models (LLMs) into human-understandable computational circuits.
Hugging Face Trending Papers
Aug 11
Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity
Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.
arXiv AI
Jul 17
Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models
arXiv:2607. 14703v1 Announce Type: cross Abstract: Multiple instance learning (MIL) has become the main paradigm for whole-slide image (WSI) analysis in computational pathology.
arXiv AI
Jun 3
Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction
arXiv:2601. 11667v2 Announce Type: replace-cross Abstract: Transformer architectures deliver state-of-the-art accuracy via dense full-attention, but their quadratic time and memory complexity with respect to sequence length limits practical deployment.
arXiv Machine Learning
Jul 2
Information-Regularized Attention for Visual-Centric Reasoning
arXiv:2607. 00434v1 Announce Type: cross Abstract: Vision-language models (VLMs) have become a paradigm for multimodal learning, yet remain unstable due to object hallucination, weak visual grounding, and catastrophic forgetting after full-parameter instruction tuning.