arXiv AI
OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jul 10
Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation
arXiv:2604. 10180v2 Announce Type: replace-cross Abstract: Disaggregation maps parts of an AI workload to different types of GPUs, offering a path to utilize modern heterogeneous GPU clusters.
arXiv AI
Jun 16
Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
arXiv:2605. 21312v2 Announce Type: replace-cross Abstract: Modern LLM serving is no longer homogeneous or monolithic.
arXiv AI
Jul 13
Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations
arXiv:2607. 09172v1 Announce Type: cross Abstract: Large Language Models are reshaping how software is developed and maintained.
arXiv Machine Learning
5d ago
HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications
HoliBench is a modular benchmarking and deployment toolkit that jointly measures accuracy, latency, and energy for foundation models across a wide range of devices, from single-board computers to GPU servers. It provides a platform abstraction layer that calibrates cross-device measurements and supports multiple model modalities, inference engines, and quantization levels. Using HoliBench, the authors evaluated 20 models on 7 device types, revealing tradeoffs such as limited latency gains from quantization on low‑precision hardware and diminishing accuracy returns relative to energy consumption, while also showing that single-model profiles can predict multi-model pipeline performance within a few percent.
arXiv AI
Jun 3
FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUs
arXiv:2506. 01969v3 Announce Type: replace-cross Abstract: Efficient inference of Multi-Head Latent Attention (MLA) is challenged by deploying the DeepSeek-R1 671B model on a single Multi-GPU server.
arXiv AI
Jun 2
Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
arXiv:2603. 29002v3 Announce Type: replace-cross Abstract: Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning.