arXiv:2506. 01260v2 Announce Type: replace Abstract: Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks.
By Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham, Yan Zuo, Alexander Long
arXiv:2606. 16384v1 Announce Type: new Abstract: Pretraining language models with extended context windows enhances their ability to leverage rich information during generation.
By Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi, Gil Avraham, Violetta Shevchenko, Yan Zuo, Chamin Hewa Koneputugodage, Alexander Long
Deep neural networks have witnessed remarkable advancements in recent years and have become integral to various applications. However, alongside these developments, training and deployment of neural network models on embedding and edge devices face significant challenges due to limited memory and computational resources.
arXiv:2608. 06441v1 Announce Type: new Abstract: Full-graph GNN training delivers high accuracy but scales poorly on multi-server clusters due to heavy, irregular inter-node embedding exchanges.
By Guofan Yu, Sitian Chen, Zhenheng Tang, Xiaowen Chu, Amelie Chi Zhou
arXiv:2607. 17913v1 Announce Type: cross Abstract: Distributed Fine-Tuning (DFT) of large-scale Foundation Models (FMs) on resource-constrained edge devices is limited by local compute constraints and communication overhead.
By Bas Meuwissen, Vasileios Tsouvalas, Nirvana Meratnia
The paper introduces the Linear Reusable Neural Bases Architecture (LRNBA), a framework that represents each network block as a linear combination of shared neural bases to improve parameter efficiency and reduce memory cost. Inspired by recurrent neural network designs, LRNBA enables the construction of wider and deeper networks within the same parameter budget. Experiments show that models using LRNBA converge as fast or faster than classical architectures, achieve lower loss, and maintain stable training dynamics.
By Binshuai Wang