← Back to all news
arXiv Machine Learning By Zhongkai Yu, Haotian Ye, Chenyang Zhou, Ohm Rishabh Venkatachalam, Zaifeng Pan, Zhengding Hu, Junsung Kim, Won Woo Ro, Po-An Tsai, Shuyi Pei, Yangwook Kang, Yufei Ding

AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving

Read the original on arXiv Machine Learning →

AMMA is a multi‑chiplet, memory‑centric architecture designed to reduce latency in large‑context attention for language models. It replaces GPU compute dies with HBM‑PNM cubes, doubling memory bandwidth and employing a logic‑die microarchitecture, two‑level hybrid parallelism, and a reordered collective flow to minimize communication overhead. Design‑space exploration and evaluations show AMMA achieves 15.5× lower attention latency and 6.9× lower energy consumption compared with the NVIDIA H100.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jul 30

LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving

arXiv:2607. 26491v1 Announce Type: cross Abstract: The energy consumption of Large Language Model (LLM) serving is becoming a major system challenge as deployment scales, driven by hardware power and thermal constraints and rising electricity costs.

By Ming-Yen Lee, Hanchen Yang, Faaiq Waqar, Harsono Simka, Tushar Krishna, Muhammed Ahosan Ul Karim, Shimeng Yu
llms
More like this →
arXiv AI
Oct 27

SPARTA: A Sparse Ternary LLM Accelerator with Adaptive Attention and Unified Datapath for Long-Context Edge Inference

SPARTA is a sparse ternary LLM accelerator designed for long‑context inference on edge devices. It combines a phase‑aware sparse dataflow, a unified compute fabric that maps ternary SpMM/SpMV and bit‑serial attention onto shared processing elements, and an adaptive two‑stage attention engine that reduces KV‑cache traffic. The design achieves significant reductions in KV traffic (4.1‑4.8× over full attention) and outperforms NVIDIA Jetson AGX Orin and AMD Ryzen 7 9700X in generation throughput and energy efficiency, while also surpassing other accelerators in throughput, energy, and area efficiency.

By Yanzhen Zhou, Zhengyu Han, Qunyou Liu, Zheyu Yan, Weihong Xu, Cheng Zhuo
llmsefficiencybenchmarks
More like this →