SOLAR: SVD-Optimized Lifelong Attention for Recommendation
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
The Flow has not summarised this story yet — read it at arXiv Computer Vision.
SequenceO1 is an end‑to‑end framework that enables ultra‑long (up to 100K interactions) sequence modeling for recommendation systems. It compresses raw user histories into a fixed‑size sketch using Sketch Attention and then models short‑term and long‑term interests with Target‑to‑History Cross Attention. The system incorporates low‑rank caching, batching, pipeline lift, and a FlashSA kernel to keep training and inference efficient, achieving consistent offline and online performance gains when deployed at full traffic on Douyin.
arXiv:2609.13141v1 Announce Type: new Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context u...
LIME is a new recommendation system architecture that tackles the high computational cost of transformers by introducing low‑rank link embeddings and a linear attention mechanism called LIME‑XOR. These innovations decouple user and candidate interactions, allowing pre‑computation of attention weights and reducing inference complexity to be nearly independent of candidate set size and linear in user sequence length. Experiments on public and industrial datasets demonstrate that LIME matches transformer performance while achieving a tenfold speedup on large candidate sets or long sequences, and it has been shown to improve user engagement on a major recommendation platform.
arXiv:2607. 24025v1 Announce Type: cross Abstract: Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models.
Adamas is a token‑level sparse attention mechanism that uses the Hadamard transform, bucketization, and 2‑bit compression to create compact representations, followed by lightweight distance‑based top‑k selection for efficient long‑context inference. It achieves near‑lossless performance on Llama‑3.1‑8B‑Instruct and delivers up to 6.5× speedup in self‑attention and 1.7× in end‑to‑end decoding at a 32K context length. Adamas also matches full attention performance on challenging long‑form reasoning benchmarks such as AIME 2024 and 2025.
arXiv:2607. 27692v1 Announce Type: cross Abstract: Top-$K$ sparse attention reduces the cost of Softmax and value aggregation by attending to only a small subset of key--value (KV) entries.