Shiva-DiT: Residual-Based Differentiable Top-$k$ Selection for Efficient Diffusion Transformers

Shiva-DiT introduces a Residual-Based Differentiable Top‑k selection mechanism for Diffusion Transformers, enabling hard top‑k token selection in the forward pass while using a residual‑aware straight‑through estimator to propagate gradients to token scores and the budget k. A Context‑Aware Router and Adaptive Ratio Policy learn layer‑ and timestep‑dependent retention schedules to meet a target average budget. Experiments on SD3‑Medium, Flux.1‑dev, and PixArt‑Σ demonstrate consistent reductions in FLOPs and latency, with Shiva‑DiT offering four fidelity‑latency operating points and achieving a 1.54× wall‑clock speedup on SD3‑Medium while maintaining competitive fidelity.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• diffusion
• efficiency