Temporal-Attention Head Specialization During Video Diffusion Training
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
The Flow has not summarised this story yet — read it at arXiv AI.
arXiv:2603.17825v2 Announce Type: replace Abstract: In this work, we study the role of Massive Activations (MAs), which are rare, high-magnitude spikes confined to a few fixed hidden dimensions in vi...
arXiv:2605. 14513v2 Announce Type: replace-cross Abstract: Sparse attention accelerates video diffusion by allowing each attention head to focus on only a small subset of interactions.
Focused Forcing is a training‑free method for selecting key‑value (KV) cache entries in autoregressive video diffusion models. It selects historical frames per generated frame by combining attention and diversity scores, and allocates larger budgets to more important heads. The approach yields up to 1.48× faster generation while improving visual quality and text alignment across several generation paradigms.
arXiv:2609.08505v1 Announce Type: cross Abstract: Reliable video generation requires more than high-quality frames to form a coherent story: a model must maintain a persistent state, transporting vis...
arXiv:2601. 11641v3 Announce Type: replace-cross Abstract: While Diffusion Transformers (DiTs) have achieved notable progress in video generation, this long-sequence generation task remains constrained by the quadratic complexity inherent to self-attention mechanisms, creating significant barriers to practical deployment.
arXiv:2610.29441v1 Announce Type: new Abstract: Full 3D attention is costly in video Diffusion Transformers. Block-sparse attention reduces this cost by computing only selected attention blocks. Howe...