The paper introduces Library of Layers (LoL), an architecture that dynamically orders transformer layers during inference, allowing a single model to adapt its computation to the task at hand. LoL’s single hyperparameter—the size of its global library of transformer blocks—enables a smooth transition between parameter reuse and sparsity, unifying dense, depth‑recurrent, and mixture‑of‑experts trade‑offs. Experiments show LoL matches or exceeds these baselines on text, synthetic Sudoku and maze tasks, and demonstrates task‑specific routing in multilingual and safety‑focused settings.
arXiv:2608. 15360v1 Announce Type: cross Abstract: While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing.
arXiv:2410. 13077v2 Announce Type: replace-cross Abstract: Transformer-based Large Language Models (LLMs) traditionally rely on final-layer loss for finetuning and final-layer representations for predictions, potentially overlooking the predictive power embedded in late layers.
arXiv:2601. 07372v2 Announce Type: replace-cross Abstract: While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation.
arXiv:2603.21676v2 Announce Type: replace-cross Abstract: Standard Transformers have a fixed computational depth, limiting their ability to generalize to tasks that require variable-depth reasoning....
The paper introduces domain-specific languages (DSLs) as an explicit representation layer to bridge the gap between high-level natural-language abstractions and concrete actions. Experiments on the visually-grounded “Hexagons” task and the “Text2CAD” benchmark show that code generation using a human-curated DSL yields dramatic performance gains, especially at higher abstraction levels, while automatically induced DSLs only partially recover these gains. The study highlights that the right representation can close much of the performance gap caused by abstraction, but that automatically learning such representations remains a challenge.
arXiv:2609.39967v1 Announce Type: cross Abstract: Recursive reasoning models apply a small shared Transformer block many times to refine a latent state. This gives them large effective depth with few...
arXiv:2609.15131v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) require substantial computation to process numerous visual tokens across all transformer layers. Most method...
arXiv:2606. 06574v1 Announce Type: new Abstract: Large language models (LLMs) perform inference by following a fixed depth and order, non-recurrent execution of all layers.
arXiv:2601. 21461v3 Announce Type: replace-cross Abstract: Modern sparse language models typically achieve sparsity through Mixture-of-Experts (MoE) layers, which dynamically route tokens to dense MLP "experts.
arXiv:2610.23455v1 Announce Type: cross Abstract: LLMs and LVLMs are increasingly deployed across heterogeneous hardware, ranging from data-center GPUs to resource-constrained edge devices. Their fix...
arXiv:2606. 16825v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory.
arXiv:2610.34756v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) has emerged as the dominant paradigm for parameter-efficient fine-tuning of large language models. Standard all-layer LoRA...