← Back to all news
arXiv Computation and Language By Li-Ming Yu, Hongyu Zhou

GatedV: Self-Gated Values for Transformer Language Models

Read the original on arXiv Computation and Language →

The paper introduces GatedV, an attention variant that applies the SiLU activation only to the value matrix before aggregation, adding no extra trainable parameters. Experiments on a decoder‑only Transformer show that, with matched parameter counts and training budgets, GatedV consistently lowers held‑out perplexity—by 2.0% at 2.33M parameters and 2.4% at 26.16M parameters—across multiple seeds. Ablations indicate that the performance gains are not fully captured by simple affine rescaling of values, and a benchmark on NVIDIA DGX Spark reports a modest 0.7% increase in full‑context forward latency for the larger model.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 12

MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution

MS-GLA introduces Multi-Scale Gated Linear Attention, distributing attention heads across multiple temporal resolutions to address representational bottlenecks in GLA Transformers. Coarser resolutions pool longer token spans for long‑range dependencies, while finer heads capture local syntax, and a learnable fusion layer recombines outputs dynamically. Experiments on language modeling, recall‑intensive tasks, and long‑context generalization show MS‑GLA outperforms GLA with up to 18.9% improvement on recall tasks and 9.5% lower perplexity on language modeling benchmarks.

By Prasoon Dev, Anirudh Sankar, Vasudeva Varma
llmsbenchmarks
More like this →
arXiv AI
Oct 12

Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers

Orthros introduces a phase‑aware heterogeneous attention mechanism that dynamically switches between linear‑complexity prefilling and full‑attention decoding, reducing the asymptotic cost compared to standard Transformers in long‑context scenarios. The method supports efficient Transformer‑to‑Orthros conversion, achieving higher performance than the original model and delivering a 21.9× prefilling speedup while maintaining strong in‑context recall, including 128K‑scale needle‑in‑a‑haystack tasks. These results demonstrate Orthros’s potential as a more efficient and flexible alternative for industrial Transformer applications.

By Yuhong Chou, Zehao Liu, Yuqi Pan, Qian Liu, Xianwei Chen, Zhongzhu Zhou, Jibin Wu
llms
More like this →