PISA introduces a training‑free Piecewise Sparse Attention mechanism that preserves full attention span while reducing complexity. It keeps exact computation for critical blocks and approximates non‑critical ones using block‑wise Taylor expansion, avoiding the quality loss seen in conventional sparse attention. Experiments show significant speedups—up to 2.57× on Hunyuan‑Video—while maintaining or improving visual quality across video and image generation tasks.
By Haopeng Li, Shitong Shao, Wenliang Zhong, Zikai Zhou, Lichen Bai, Hui Xiong, Zeke Xie
arXiv:2605. 14513v2 Announce Type: replace-cross Abstract: Sparse attention accelerates video diffusion by allowing each attention head to focus on only a small subset of interactions.
By Xuzhe Zheng, Yuexiao Ma, Jing Xu, Xiawu Zheng, Rongrong Ji, Fei Chao
iCATS introduces a training‑free sparse attention mechanism for Diffusion Transformers that accelerates video generation by reducing computations. It improves importance estimation by clustering query‑key dot‑product interactions and reformulates the objective as a quadratic form for low‑cost computation. The method also adapts sparsity dynamically across denoising timesteps using an SNR‑guided schedule and employs a tail‑merging strategy to enhance GPU kernel utilization, achieving significant speedups while maintaining high PSNR scores.
By Chengfeng Han, Baole Ai, Xianlu Bian, Jie Yao, Zilong Huang, Ang Wang, Dandan Ding
arXiv:2609.37001v1 Announce Type: cross
Abstract: Diffusion Transformers (DiTs) enable high-quality video generation but suffer from substantial inference latency, primarily attributable to the compu...
By Xingyu Jia, Baole Ai, Ang Wang, Kang Zhao, Yong Li
arXiv:2608. 12032v1 Announce Type: cross Abstract: Video diffusion transformers are costly to sample: every denoising step applies self-attention over a long 3D token sequence, a quadratic cost that dominates as resolution and duration grow.
By Enhuai Liu, Yunke Wang, Yutong Wang, Changming Sun, Chang Xu
arXiv:2610.29441v1 Announce Type: new
Abstract: Full 3D attention is costly in video Diffusion Transformers. Block-sparse attention reduces this cost by computing only selected attention blocks. Howe...
By Tianze Xia, Xiaoling Zhou, Ruya Lan, Yiqiao Liao, Chengru Song, Shiming Xiang
Diffusion Transformers (DiTs) incur quadratic self-attention cost over spatiotemporal tokens. Existing training-free sparse attention methods often construct sparse masks from block-level or cluster-level proxy scores, which can obscure fine-grained differences among keys and miss high contribution keys under aggressive sparsity.
The paper introduces Growing Sparsity and Reduced Search (GSRS), a training‑free framework that accelerates Diffusion Transformers for video generation by progressively increasing sparsity and reusing early attention blocks to limit search space. GSRS uses Top‑p selection for adaptive sparsity and a precise evaluation scheme that computes exact attention weights with negligible overhead. Experiments on HunyuanVideo and Wan 2.1 show 1.55×–2.21× speedups while preserving generation quality comparable to full attention.
By Yun Dai, Yuyun Chen, Jiarui Wen, Huiping Zhuang, Cen Chen, Ziqian Zeng
arXiv:2602. 01801v2 Announce Type: replace-cross Abstract: Autoregressive video diffusion models enable streaming generation, opening the door to long-form synthesis, video world models, and interactive neural game engines.
By Dvir Samuel, Issar Tzachor, Matan Levy, Michael Green, Gal Chechik, Rami Ben-Ari
arXiv:2607. 03012v1 Announce Type: cross Abstract: Video Diffusion Transformers (VDiTs) have demonstrated significant capabilities in high-fidelity video generation.
By Dongyeun Lee, Amir Zandieh, Vahab Mirrokni, Junmo Kim, Insu Han
The paper introduces SparsePR, a training‑free block‑sparse attention method for video transformers that partitions query‑key responses and reconstructs the residual via probe‑fitted affine corrections. By pairing sampled‑query key responses into K/V groups and using centroids to guide shared routing, SparsePR reduces attention‑reconstruction error across diverse video generation and world‑model tasks. Experiments show consistent error reductions, with probe fitting contributing most of the improvement, while maintaining generation quality at 22.0–26.0% executed‑pair density and delivering 1.48×–2.61× speedups.
By Pardis Taghavi, Reza Langari, Gaurav Pandey
arXiv:2609.15810v1 Announce Type: new
Abstract: Diffusion Transformers deliver state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant deployment cost,...
By Xingyang Li, Dongyun Zou, Shining Zhang, Jiacheng Chen, Haocheng Xi, Lvmin Zhang, Jun-Yan Zhu, Song Han, Zhekai Zhang, Yujun Lin, Muyang Li