arXiv Computation and Language October 9, 2026 By Mingzi Cao, Xi Wang, Nikolaos Aletras

Optimal Transport Depth Up-Scaling

Read the original on arXiv Computation and Language →

Optimal Transport Depth Up-Scaling (OT‑DUS) is a method for expanding large language models by inserting new layers into a pre‑trained model instead of training from scratch. It uses Optimal Transport theory to align and fuse functionally corresponding neurons across adjacent base layers, avoiding the neuron permutation mismatch that plagues existing depth‑up‑scaling techniques. Experiments show OT‑DUS outperforms other methods in both general and specialized domains, and inserting new layers at higher positions improves both performance and training efficiency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Jun 2

DOT-MoE: Differentiable Optimal Transport for MoEfication

arXiv:2606. 01666v1 Announce Type: cross Abstract: The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency.

By Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta

llms efficiency benchmarks

arXiv AI

Jun 2

Inverse Depth Scaling From Most Layers Being Similar

arXiv:2602. 05970v2 Announce Type: replace-cross Abstract: Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies.

By Yizhou Liu, Sara Kangaslahti, Ziming Liu, Jeff Gore

llms safety

arXiv AI

Aug 17

Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

arXiv:2608. 13596v1 Announce Type: cross Abstract: Heterogeneous model fusion seeks to combine models that differ in tasks, initializations, architectures, or scales.

By Jiahe Fan, Si Chen, Yinghao Hou, Aiyuan Zhang, Hong Xie

llms efficiency benchmarks safety

arXiv Machine Learning

Jun 2

Scaling depth capacity via zero/one-layer model expansion

arXiv:2511. 04981v2 Announce Type: replace Abstract: Model depth is a double-edged sword in deep learning: deeper models achieve higher accuracy but require higher computational cost.

By Zhiqi Bu

arXiv AI

Oct 5

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

arXiv:2602.09009v2 Announce Type: replace-cross Abstract: Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep l...

By Yilang Zhang, Bingcong Li, Niao He, Georgios B. Giannakis

llms diffusion

arXiv AI

Jun 16

Scaling Adaptive Depth with Norm-Agnostic Residual Networks

arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.

By Tom\'as Figliolia, Beren Millidge

llms efficiency

One email a morning, machine-written

Related stories

DOT-MoE: Differentiable Optimal Transport for MoEfication

Inverse Depth Scaling From Most Layers Being Similar

Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

Scaling depth capacity via zero/one-layer model expansion

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

Scaling Adaptive Depth with Norm-Agnostic Residual Networks