arXiv Computation and Language
P2N: Reusing Deep Representations for Greater Effective Depth
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Machine Learning
Jul 24
Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
arXiv:2607. 21291v1 Announce Type: cross Abstract: Large language models (LLMs) achieve strong generation and reasoning performance, but the Transformer architecture incurs high inference cost.
arXiv AI
1d ago
Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves
The paper introduces Dreamer, a depth‑recurrent attention mixture framework that combines sequence attention, depth attention, and sparse expert attention to overcome the hidden‑size bottleneck in latent reasoning models. Dreamer decouples scaling dimensions, allowing efficient depth‑recurrent scaling, and achieves state‑of‑the‑art performance on language reasoning benchmarks while using 2 to 8× fewer training tokens than FLOP‑, parameter‑, and memory‑matched baselines. It also demonstrates significantly greater expert selection diversity, up to 11× larger than current MoE models.
arXiv Machine Learning
Oct 15
Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
arXiv:2610.19121v1 Announce Type: new Abstract: Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reason...
arXiv AI
Aug 5
On the Limits of Layer Pruning for Generative Reasoning in Large Language Models
arXiv:2602. 01997v3 Announce Type: replace-cross Abstract: Recent work has shown that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning.
arXiv Machine Learning
Sep 24
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.
arXiv Computation and Language
Oct 12
RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models
The paper introduces RAMP, a Recurrent Adaptive Mixed Precision method that applies low‑precision weight‑and‑activation computation early in the recurrent depth of language models and switches to high precision for final refinement. By analyzing how quantization errors propagate, the authors show that early‑stage errors are dampened by subsequent steps, allowing RAMP to maintain near‑full‑precision prefill quality. Experiments up to 4K context lengths demonstrate up to a 1.16× speedup and a 13.9% latency reduction on an NVIDIA H100.