arXiv Machine Learning
DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 28
MMOE: Modernizing Diffusion Transformers with Efficient Expert Design
arXiv:2607. 24665v1 Announce Type: cross Abstract: Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows.
arXiv AI
Jun 26
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
arXiv:2606. 26538v1 Announce Type: cross Abstract: Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value.
arXiv AI
Sep 15
LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference
LayerRoute is a parameter‑efficient technique that enables adaptive skipping of transformer layers in large language models. It adds a lightweight per‑layer router (~21.5K parameters) and LoRA adapters (rank 8, ~1.08M parameters) to each of the 24 blocks in Qwen2.5‑0.5B‑Instruct, training them jointly with a gate‑regularized language‑modeling objective. Across ten independent runs, the method consistently identifies nine middle layers as skip‑eligible, achieves a verified wall‑clock speedup of 1.02x–1.06x, and improves perplexity by an average of 1.16 points, while the router’s decisions vary per input, confirming genuine adaptive behavior.
arXiv Machine Learning
Oct 15
Your Transformer Is Secretly Elastic
The paper presents a retraining‑free technique that transforms pre‑trained transformers into elastic models capable of adapting to any computational budget on the fly. By jointly ranking every MLP neuron and attention head on a common scale and applying soft pruning across multiple sparsity targets, the method achieves smooth performance degradation up to 60% sparsity while outperforming one‑shot baselines. Experiments on 27 vision and text transformer encoders show significant parameter reductions with minimal accuracy loss and notable throughput and latency gains on GPUs, all produced in minutes.
arXiv Machine Learning
Jul 21
Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
arXiv:2607. 18130v1 Announce Type: new Abstract: Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections.
arXiv Machine Learning
Aug 4
Geometry-Guided Layerwise FFN Width Allocation in Transformers
arXiv:2608. 02064v1 Announce Type: new Abstract: Feed-forward networks (FFNs) account for a large fraction of Transformer parameters, yet their hidden width is usually constant across depth.