← Back to all news
arXiv Machine Learning By Adrian Goldwaser, Michael Munn, Javier Gonzalvo, Benoit Dherin

Equivalence of Context and Parameter Updates in Modern Transformer Blocks

Read the original on arXiv Machine Learning →

arXiv:2511. 17864v3 Announce Type: replace Abstract: Recent research has established that the impact of context in a vanilla transformer can be represented implicitly by forming a token-dependent, rank-1 patch to its MLP weights.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 22

Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers

The paper introduces CS-MoE, a Transformer architecture that shares experts across layers to reduce inter‑layer parameter redundancy. By combining layer‑independent experts with a globally shared expert pool, CS‑MoE allows elastic control over token‑level parameter activation and computational cost. Experiments show that CS‑MoE achieves lower perplexity than equal‑scale dense Transformers while activating only 55% of parameters, and its performance scales with the number of activated experts, approaching MoE performance within a fixed FLOPs budget.

By Dian Jiao, Jiaxin Duan, Shuai Zhao, Jiabing Leng, Yiran Zhang, Feng Huang
llms
More like this →
arXiv Machine Learning
Jul 3

Hyperloop Transformers

arXiv:2604. 21254v3 Announce Type: replace Abstract: LLM architecture research generally aims to maximize model quality subject to fixed compute/latency budgets.

By Abbas Zeitoun, Lucas Torroba-Hennigen, Yoon Kim
llmsefficiency
More like this →
arXiv Machine Learning
1d ago

Learning Rate Transfer for Hybrid Transformer-SSM Architectures

arXiv:2610. 01172v1 Announce Type: new Abstract: We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models.

By Jimin Seo, Gyubok Lee, Yeonsik Jo, Kiwoong Yoo, Yeongoon Kim, Minhae Oh, Jin Woo Koo, Suhwan Kim, Nakyung Lee, Minsik Seol, Idris Nechnech, Jaehyeon Kim, Giho Lee, Jungwoo Lee
llms
More like this →
arXiv Machine Learning
Jun 2

From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction

arXiv:2511. 12081v2 Announce Type: replace-cross Abstract: Despite massive investments in scale, deep models for click-through rate (CTR) prediction often exhibit rapidly diminishing returns -- a stark contrast to the {predictable scaling laws} seen in large language models (LLMs).

By Bencheng Yan, Yuejie Lei, Zhiyuan Zeng, Zheye Deng, Di Wang, Kaiyi Lin, Pengjie Wang, Chuan Yu, Jian Xu, Bo Zheng
llmsbenchmarkssafety
More like this →