arXiv AI
Sep 10

SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching

SequenceO1 is an end‑to‑end framework that enables ultra‑long (up to 100K interactions) sequence modeling for recommendation systems. It compresses raw user histories into a fixed‑size sketch using Sketch Attention and then models short‑term and long‑term interests with Target‑to‑History Cross Attention. The system incorporates low‑rank caching, batching, pipeline lift, and a FlashSA kernel to keep training and inference efficient, achieving consistent offline and online performance gains when deployed at full traffic on Douyin.

By Lin Guan, Jia-Qi Yang, Zhishan Zhao, Jiaqi Huang, Hangyu Wang, Longbin Li, Beichuan Zhang, Haonan Jiang, Jinan Ni, Xiangyu Fan, Xiaowen Li, Ziyao Ren, Yuhang Qi, Xiaolong Zhu, Xuanyuan Luo, Qiwei Chen, Yi Cheng, Lele Yu
arXiv Machine Learning
Oct 9

LIME: Link-based User-item Interaction Modeling with Decoupled XOR Attention for Efficient Test Time Scaling

LIME is a new recommendation system architecture that tackles the high computational cost of transformers by introducing low‑rank link embeddings and a linear attention mechanism called LIME‑XOR. These innovations decouple user and candidate interactions, allowing pre‑computation of attention weights and reducing inference complexity to be nearly independent of candidate set size and linear in user sequence length. Experiments on public and industrial datasets demonstrate that LIME matches transformer performance while achieving a tenfold speedup on large candidate sets or long sequences, and it has been shown to improve user engagement on a major recommendation platform.

By Yang Liu, Yunjiang Jiang, Ayush Agarwal, Yihan Wu, Haoran Liu, Bi Xue
arXiv Machine Learning
Jul 28

SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation

arXiv:2607. 24025v1 Announce Type: cross Abstract: Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models.

By Yu Cui, Yi Xu, Jiahao Wang, Hao Zhang, Yu Zhang, Xiaoyi Zeng, Can Wang, Jinxin Hu, Jiawei Chen
arXiv Computation and Language
Nov 9

Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference

Adamas is a token‑level sparse attention mechanism that uses the Hadamard transform, bucketization, and 2‑bit compression to create compact representations, followed by lightweight distance‑based top‑k selection for efficient long‑context inference. It achieves near‑lossless performance on Llama‑3.1‑8B‑Instruct and delivers up to 6.5× speedup in self‑attention and 1.7× in end‑to‑end decoding at a 32K context length. Adamas also matches full attention performance on challenging long‑form reasoning benchmarks such as AIME 2024 and 2025.

By Siyuan Yan, Guo-Qing Jiang, Yuchen Zhang, Xiaoxing Ma, Ran Zhu, Chun Cao, Jingwei Xu