The paper introduces Multi-Stream Attention Residuals (MSAR), an extension of Attention Residuals that applies multi-head depth attention inspired by multi-head sequence attention. MSAR uses multi-scale causal convolutions to provide diverse contextual views for each head, then adaptively mixes the retrieved streams into a single-stream input, avoiding persistent multi-stream storage. Across 12 downstream benchmarks at 18B and 28B MoE scales, MSAR outperforms prior residual variants and achieves a 1.58× compute speedup over the standard residual baseline.
By Xiaohan Qin, Xiangdong Zhang, Tuo Dai, Sunan Zou, Huaijin Wu, Yebin Yang, Zhuo Xia, Lin Yao, Yuliang Liu, Junchi Yan
Multi-Stream Attention Residuals (MSAR) extends the Attention Residuals (AttnRes) mechanism by introducing multi-head depth attention, inspired by multi-head attention along the sequence axis. MSAR generates diverse streams from sequence context via multi-scale causal convolutions, then adaptively mixes them to produce a single-stream layer input, enabling multi-stream computation without extra storage or communication. Across 12 downstream benchmarks at 18B and 28B MoE scales, MSAR consistently outperforms prior residual variants and achieves a 1.58× iso-loss compute speedup over the standard residual baseline.
The paper introduces CATTLE, a transfer learning framework for disjoint tabular datasets that eliminates the need for shared features by leveraging generalized context learned through transformer projection weights. By using key, value, and query weights from source and target domains, CATTLE performs cross‑domain attention transfer in a data‑agnostic manner. Experiments on ten source‑target pairs demonstrate that CATTLE outperforms nine state‑of‑the‑art baselines, achieving the best average rank (2.9) and a 3.7% AUROC improvement.
By Kazi F. Akhter, Ibna Kowsar, Manar D. Samad
arXiv:2607. 27303v1 Announce Type: new Abstract: Temporal heterogeneous graphs offer a natural abstraction for dynamic relational systems in which diverse node and relation types co-exist and evolve over time.
By Yixin Peng, Diego Collarana, Er Jin, Stefan Decker
The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.
By Ke Wan, Chen Chen
arXiv:2512. 12477v2 Announce Type: replace Abstract: Estimating node importance in heterogeneous knowledge graphs is a fundamental problem underlying recommendation, search, and knowledge decision systems.
By Jiawen Chen, Yanyan He, Qi Shao, Mengli Wei, Duxin Chen, Wenwu Yu, Yanlong Zhao