The paper compares two strategies for handling memory limits in large language model (LLM) serving: tensor parallelism, which distributes weights and KV cache across multiple GPUs, and KV compression, which reduces cache size via quantisation and eviction on a single GPU. Using a cost‑normalised simulator calibrated on A100, A40, and H100 hardware, the authors find that across two models (Llama‑2 7B and 70B) and various GPU configurations, compression consistently outperforms tensor parallelism in cost per million tokens, offering 1.20× to 2.00× savings. The study identifies a model‑size threshold (~36B parameters on an 80 GB card) where compression dominates, while tensor parallelism becomes necessary only for larger models where weights alone exceed a single GPU’s capacity.
By Srikanta Datta Tumkur, Mehar Simhadri, Anshu Bansal, Jay Iyer, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly, Raj Dandekar
SenseShift is an encoder-based framework that enables fine‑grained, sentence‑level sentiment control in text generation. It uses bidirectional attention, quantized sentiment signals, and iterative mask infilling to generate local sentences conditioned on target sentiment intensity. Experiments on story and review generation show that SenseShift delivers stronger sentiment controllability while preserving text quality and robustness to out‑of‑domain inputs compared to larger decoder‑based baselines.
By Shahed Masoudian, Markus Frohmann, Emmanouil Karystinaios, Navid Rekabsaz, Markus Schedl
arXiv:2608.24549v1 Announce Type: new
Abstract: Persistent entropy is the Shannon entropy of a persistence-based probability measure defined on a persistence diagram. However, its cross-entropy versi...
By Sijin Yeom, Jae-Hun Jung
The paper introduces an elastic key‑value (KV) cache for large language model (LLM) serving that dynamically reclaims a pre‑allocated reserve during decode‑heavy phases and restores it before prefill, using a userspace CUDA virtual‑memory trick that requires no driver changes. The authors implement this mechanism, test it under realistic workloads, and find that it offers only marginal benefits—about a 1 % difference in time‑to‑first‑token for large prefill chunks—and that simpler strategies such as lowering the maximum batch size can achieve similar results. The study also notes that the reserve’s impact diminishes with higher tensor‑parallelism levels.
whyItMatters":"The work demonstrates that a dynamic KV cache reclamation strategy can be implemented without driver patches and that its practical benefits are limited, guiding future LLM serving optimizations toward simpler approaches."
By Sathishkumar Sivashanmugam
Masked diffusion language models (dLLMs) promise faster text generation by denoising multiple tokens simultaneously, yet their real‑world serving behavior has been largely unexamined. Using LLaDA‑8B‑Instruct on a single NVIDIA H200 GPU, the study finds that request difficulty is discretized into 11 step‑count levels, short‑budget benchmarks underestimate serving variance, and only 24% of single‑request time is GPU computation, with batching mainly reducing CPU dispatch overhead. The authors also demonstrate that output quality remains stable across batch sizes and propose a batch‑timeout rule for synchronized batching under Poisson arrivals.
By Farhana Amin, Sabiha Afroz, Mona Moghadampanah, Dimitrios S. Nikolopoulos
The Transformer Accelerator (TFA) is a synthesizable, parameterizable INT8 memory‑to‑memory engine designed for transformer inference and machine translation. It features a one‑time‑multiplexed datapath that handles prompt processing and autoregressive generation, and implements key operations such as matrix multiplication, softmax, RMSNorm, and elementwise functions through eight 512‑bit macro‑op descriptors. In extensive verification, TFA achieved zero mismatches across 25 tests and 34 constrained‑random runs, matched floating‑point references on multiple translation tasks, and delivered a 20× speedup over a 22‑thread CPU while projecting significant energy reductions in larger designs.
By Shashank
arXiv:2608.23843v1 Announce Type: new
Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
By Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li
arXiv:2608.24207v1 Announce Type: new
Abstract: Semantic identifiers (SIDs) represent entities as hierarchical token sequences for generative retrieval and recommendation. Residual-quantization token...
By Yunxiao Luo, Siyuan Wang, Ben Chen, Chenyi Lei
The paper introduces the "Compression Trinity," a unified framework that jointly applies sparsity, quantization, and low‑rank approximations to compress large language models. It presents several methods—MKOR, SLoPe, OPTIMA, PATCH, and SLiM—that leverage these three pillars to accelerate training, reduce memory bandwidth, and recover accuracy, achieving significant speedups and accuracy gains over existing techniques. The results demonstrate that combining all three compression strategies is essential for efficient, scalable, high‑performance LLM deployment.
By Mohammad Mozaffari
QABBA is a quantized variant of the Adaptive Brownian Bridge-based Aggregation (ABBA) method that compresses long numerical time‑series into short symbolic sequences using integer‑quantized symbolic centers. It reduces parameter storage and enables integer arithmetic while providing rigorous error bounds—including a dimension‑free segment error, a time‑domain reconstruction error, a stability condition for symbolic assignment, and a bit‑allocation rule. The resulting symbolic strings can be fed directly into pretrained large language models, and experiments on several benchmark archives show a practical trade‑off among storage, reconstruction accuracy, and downstream predictive performance.
By Erin Carson, Xinye Chen, Fei He, Cheng Kang
Polynomial-Augmented Neural Networks (PANNs) merge deep neural networks with polynomial expansions to leverage the flexibility of DNNs and the rapid convergence of polynomials. The architecture introduces orthogonality constraints, basis pruning, and polynomial preconditioning to stabilize training and improve accuracy across diverse problems. Experiments show that PANNs outperform both pure DNNs and polynomial methods in approximating smooth and limited‑smoothness functions, as well as in solving partial differential equations.
By Madison Cooley, Shandian Zhe, Robert M. Kirby, Varun Shankar
Edge LLM deployment is shaped by more than model size and precision: inference backend, hardware platform, memory traffic, and power management all affect latency and efficiency. We present Hydra, a c...
Morphological transforms are long-standing tools for shape and mask processing, but the de facto reference implementation in the Python ecosystem, i.e. scipy.ndimage, is CPU-only, single-array, and th...
Reinforcement learning can align diffusion models with human preferences and task-specific objectives, but endpoint rewards do not specify how an intermediate denoising prediction should change. We in...
Post-training quantization lowers the memory footprint of Large Language Models (LLMs) and speeds up inference, which is why it is now common for on-device deployment. Most of what we know about its e...
Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for data-limited replay. Through controlled experimen...
With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especial...
End-to-end multimodal driving has progressed rapidly by fusing camera and LiDAR streams. Existing pipelines remain fragile under asymmetric sensor degradation, where either an entire modality or only...
Transformer-based object trackers are renowned for their strong performance, yet dense token processing often leads to prohibitive computational cost, limiting real-time deployment on edge devices. Wh...