arXiv Machine Learning June 3, 2026 By Oliver Sieberling, Bharat Runwal, Rameswar Panda, Yoon Kim

Dynamic Short Convolutions Improve Transformers

Read the original on arXiv Machine Learning →

arXiv:2606. 03825v1 Announce Type: new Abstract: Transformers have become the dominant architecture for large language models, largely due to the scalability and flexibility of attention, feed-forward layers, residual connections, and normalization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 28

cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs

arXiv:2607. 22577v1 Announce Type: new Abstract: Scaling large language models (LLMs) has driven their success, yet dense Transformers couple capacity and computation: every parameter is activated for every token, making training and inference costs grow linearly with model size-a critical bottleneck as models approach trillion-parameter regimes.

By Xin Yang, Yemin Wang, Mingda Liu, Letian Li, Shuaishuai Cao, Zhengxiao He, Ryan Dong

llms

arXiv Machine Learning

Sep 1

You Do Not Fully Utilize Transformer's Representation Capacity

arXiv:2502.09245v3 Announce Type: replace Abstract: In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standar...

By Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky, Viacheslav Sinii, Daniil Gavrilov

llms

arXiv Computation and Language

Sep 10

ConvMem: Convolutional Memory for Long-Context Reasoning

arXiv:2609.10441v1 Announce Type: cross Abstract: While Large Language Models (LLMs) have demonstrated impressive capabilities, they often struggle with extremely long contexts due to fixed context l...

By Hongming Zhang, Zhaozhen Gu, Fengshuo Bai, Ming Hao, Qingyang Zhang, Yuanyuan Wang, Shiyang Tang, Yanna Wang, Bo Xu

llms reinforcement-learning

Hugging Face Trending Papers

Sep 9

ConvMem: Convolutional Memory for Long-Context Reasoning

While Large Language Models (LLMs) have demonstrated impressive capabilities, they often struggle with extremely long contexts due to fixed context limits. To address this, sequential approaches like...

llms reinforcement-learning

arXiv Machine Learning

Jul 14

Controllably Efficient Language Models

arXiv:2511. 05313v2 Announce Type: replace Abstract: The substantial inference costs of attention in transformers motivated the development of efficient sequence mixers: namely sparse and sliding window attention, convolutions and linear attention.

By Jatin Prakash, Aahlad Puli, Rajesh Ranganath

llms efficiency benchmarks

arXiv AI

Aug 11

Full-bandwidth transformer

arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.

By Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, John Langford

llms rag efficiency

One email a morning, machine-written

Related stories

cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs

You Do Not Fully Utilize Transformer's Representation Capacity

ConvMem: Convolutional Memory for Long-Context Reasoning

ConvMem: Convolutional Memory for Long-Context Reasoning

Controllably Efficient Language Models

Full-bandwidth transformer