T‑RoPE introduces a time‑aware Rotary Position Embedding for sequential recommendation, replacing index‑only rotations with timestamp‑based angles, learnable temporal coefficients, multiscale frequency banks, shifted query alignment, and non‑stationary key rotation. The authors prove that standard RoPE is time‑translation invariant and cannot capture seasonal contexts, while T‑RoPE breaks this invariance while preserving the RoPE interface. Across five public benchmarks and an industrial‑scale e‑commerce dataset, T‑RoPE outperforms baselines by up to 130 % in HR@10 and delivers significant online lift in conversion and order count.
By Yang Liu, Noel Loo, Ali Khanafer, Shuying Sun, Akshay Soni, Zhong Wu, Linjun Yang
Positional encoding is a fundamental component of Transformer-based generative recommendation models, where user histories are modeled as autoregressive item sequences. Most positional encoding methods are inherited from natural language processing and mainly represent discrete item order.
The paper introduces ReST, a recommendation‑native Transformer scaling framework designed to handle noisy, irregular, and sparsely supervised user behavior sequences in production ranking. ReST employs a dual‑gated attention encoder with rotary positional and temporal embeddings, and a lightweight cross decoder that decouples heavy encoding from fast decoding, enabling efficient compute‑once, decode‑many‑times ranking. Experiments on industrial and public benchmarks show that ReST outperforms traditional Transformer blocks, achieving higher accuracy and consistent scaling across sequence length, depth, and width, and a one‑week online A/B test on a production advertising platform yielded a 1.31% AUC lift and an 11.93% increase in a core revenue metric within a 50 ms P99 latency budget.
By Jie Chen, Xiangqian Yu, Yanchao Lian, Tan Lu, Run Yang, Zhengchun Shang, Xing Wang, Cheng Chen, Ke Hu, Qiang Li, Tianjiu Yin, Xiaobing Liu
arXiv:2608. 16274v1 Announce Type: cross Abstract: Positional encoding is a fundamental component of Transformer-based generative recommendation models, where user histories are modeled as autoregressive item sequences.
By Pengfei Jia, Jingjian Wang, Jingmao Li, Ge Zhang, Feng Shi
The paper introduces DSRec, a dual‑interest sequential recommendation model that separates item representations into long‑term and short‑term semantic contexts. Long‑term embeddings capture stable preferences through historical aggregation, while short‑term embeddings focus on local session intent modulated by inter‑click time intervals. Each branch is processed by a distinct State Space Model— a full‑sequence Mamba for long‑term modeling and a time‑modulated SSM for short‑term dynamics— and a residual cross‑fusion mechanism aligns the two granularities while preserving their independence. Experiments on public benchmarks show that DSRec outperforms state‑of‑the‑art methods.
By Shuiying Liao, P. Y. Mok
arXiv:2603.02561v2 Announce Type: replace-cross
Abstract: Attention mechanism remains the defining operator in Transformers since it provides expressive global credit assignment, yet its quadratic co...
By Chenghao Zhang, Chao Feng, Yuanhao Pu, Xunyong Yang, Wenhui Yu, Xiang Li, Chunjie Chen, Kaiqiao Zhan
arXiv:2511. 17388v3 Announce Type: replace-cross Abstract: Position information is essential for language modeling.
By Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher
arXiv:2607. 24025v1 Announce Type: cross Abstract: Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models.
By Yu Cui, Yi Xu, Jiahao Wang, Hao Zhang, Yu Zhang, Xiaoyi Zeng, Can Wang, Jinxin Hu, Jiawei Chen
Large language models (LLMs) have recently emerged as powerful backbones for recommender systems by reformulating recommendation as a token-level generation task. Despite their promise, we identify a pervasive yet underexplored issue: $\textit{Length Bias}$.
arXiv:2509. 10534v3 Announce Type: replace-cross Abstract: The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where.
By Anand Gopalakrishnan, Robert Csord\'as, J\"urgen Schmidhuber, Michael C. Mozer
arXiv:2604. 25834v2 Announce Type: replace Abstract: With the rapid development of the Internet, users have increasingly higher expectations for the recommendation accuracy of online content consumption platforms.
By Wenhao Li, Zihan Lin, Zhengxiao Guo, Jie Zhou, Shukai Liu, Yongqi Liu, Chuan Luo, Chaoyi Ma, Ruiming Tang, Han Li
SequenceO1 is an end‑to‑end framework that enables ultra‑long (up to 100K interactions) sequence modeling for recommendation systems. It compresses raw user histories into a fixed‑size sketch using Sketch Attention and then models short‑term and long‑term interests with Target‑to‑History Cross Attention. The system incorporates low‑rank caching, batching, pipeline lift, and a FlashSA kernel to keep training and inference efficient, achieving consistent offline and online performance gains when deployed at full traffic on Douyin.
By Lin Guan, Jia-Qi Yang, Zhishan Zhao, Jiaqi Huang, Hangyu Wang, Longbin Li, Beichuan Zhang, Haonan Jiang, Jinan Ni, Xiangyu Fan, Xiaowen Li, Ziyao Ren, Yuhang Qi, Xiaolong Zhu, Xuanyuan Luo, Qiwei Chen, Yi Cheng, Lele Yu