Profiling in PyTorch (Part 3): Attention is all you profile

Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler

Profiling in PyTorch (Part 2): From nn.Linear to a Fused MLP

Visualize and understand GPU memory in PyTorch

OpenAI standardizes on PyTorch

We are standardizing OpenAI’s deep learning framework on PyTorch.

nanoVLM: The simplest repository to train your VLM in pure PyTorch

Accelerating PyTorch distributed fine-tuning with Intel technologies

Limitations of Normalization in Attention Mechanism

arXiv:2508. 17821v3 Announce Type: replace-cross Abstract: This paper investigates the limitations of the normalization in attention mechanisms.

LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention

arXiv:2607. 11976v1 Announce Type: new Abstract: Indexer-TopK, the operation to compute the scores and select the top-k candidates, is widely used by sparse attention kernels in large language models and vector retrieval in recommendation systems and vector databases.

Linearized 2-Simplicial Attention

arXiv:2608. 09307v1 Announce Type: new Abstract: We present a linearized form of 2-simplicial attention by rewriting the trilinear score as an inner product between a composite query and a key, so that the sum over one token axis takes the same form as ordinary softmax attention.

Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining

arXiv:2509. 10406v4 Announce Type: replace Abstract: Pretraining transformers on long sequences (entire code repositories, collections of related documents) is bottlenecked by quadratic attention costs.

Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention

arXiv:2507. 07247v2 Announce Type: replace-cross Abstract: As large language models (LLMs) and visual language models (VLMs) grow in scale and application, attention mechanisms have become a central computational bottleneck due to their high memory and time complexity.

Neural Message-Passing on Attention Graphs for Hallucination Detection

arXiv:2509. 24770v2 Announce Type: replace Abstract: Large Language Models (LLMs) often generate incorrect or unsupported content, known as hallucinations.