The paper introduces COUNT, a method that builds n‑gram embeddings without adding learned parameters by storing precomputed n‑gram counts on host memory. By weighting the language model’s own unembeddings with corpus frequencies, COUNT can be added as an extra input embedding during pretraining, improving validation loss equivalent to a 57% increase in compute for a 1B‑parameter model. Additionally, COUNT enables efficient GPU memory usage and allows domain‑specific n‑gram tables to be swapped in at inference time for training‑free adaptation.
By Ian Magnusson, Noah A. Smith, Pang Wei Koh
arXiv:2603. 29002v3 Announce Type: replace-cross Abstract: Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning.
By Zifan He, Rui Ma, Yizhou Sun, Jason Cong
PTXBench is a benchmark designed to evaluate and adapt large language models (LLMs) for GPU kernel optimization using architecture-specific PTX code. It assesses functional correctness, runtime execution of target instructions, and speedup over leading libraries on GEMM and attention workloads on H100 and B200 GPUs. The study finds uneven performance across models, especially on complex attention backward tasks, and shows that fine‑tuning Qwen3.6‑27B improves some tasks but generalization remains inconsistent.
PTXBench is a benchmark designed to evaluate and adapt large language models (LLMs) for GPU kernel optimization using architecture‑specific PTX code. It assesses functional correctness, runtime execution of target instructions, and speedup over state‑of‑the‑art libraries on GEMM and attention workloads on H100 and B200 GPUs. The study finds uneven success rates, especially on complex attention backward tasks, and shows that executing target instructions does not guarantee competitive performance, with no model consistently outperforming frontier libraries. The authors also fine‑tune Qwen3.6‑27B, noting that repair‑conditioned training improves some tasks but generalization remains inconsistent, highlighting the importance of data coverage, balance, and teacher quality.
By Genghan Zhang, Yixin Dong, Chengze Fan, Zhichen Zeng, Yueming Yuan, Shaowei Zhu, Kunle Olukotun
arXiv:2610.04204v2 Announce Type: replace
Abstract: Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature,...
By Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi Shikai Fang, Guillaume Rabusseau, Sirisha Rambhatla
arXiv:2608. 14621v1 Announce Type: cross Abstract: Long-term memory is increasingly central to LLM agents, yet memory design remains a highly coupled architecture problem: what to encode, how to store it, how to retrieve it, and how to manage it can vary substantially across tasks and backbone models.
By Lin Du, Jie Zhou, Yuxuan Cai, Kai Chen, Qin Chen, Xin Li, Bo Zhang, Wei Li, Liang He
Clean is a memory‑efficient optimizer that uses Nyström sketching to approximate SOAP’s curvature preconditioners, reducing memory complexity from quadratic to linear while preserving rich curvature information. It also reintegrates off‑subspace components and offers a low‑precision variant, Q‑Clean, that cuts optimizer memory by over 50% compared to Muon while maintaining strong performance. Clean outperforms AdamW in wall‑clock time and enables pre‑training a 13B‑parameter model on a single 80GB GPU.
By Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi, Shikai Fang, Guillaume Rabusseau, Sirisha Rambhatla
arXiv:2603. 17484v2 Announce Type: replace-cross Abstract: Language models struggle to generalize beyond pretraining context lengths, limiting long-horizon reasoning and retrieval.
By Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato, Elvis Nunez, Matthew Trager, Wei Xia, Stefano Soatto
arXiv:2608. 15360v1 Announce Type: cross Abstract: While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing.
By Mohammad Aref Jafari-Raddani, Morteza Mohajjel Kafshdooz
The paper introduces LARS (Low-memory Activation-Rank Subspace), a parameter-efficient fine-tuning method that reduces the memory footprint of adapter activations by operating in a sequence-level low-rank subspace. By replacing token-level activations with pooled feature extraction, low-rank modulations, and residual projection, LARS cuts the stored activation intermediates from O(BSRL) to O(BRL). Experiments show that LARS achieves comparable performance to LoRA while lowering GPU memory usage by 23%, CPU memory usage by 47.8%, and reducing CPU power and energy consumption per optimization step by about 22%.
By Irene Tenison, Stella Ahn, Miriam Kim, Ebtisam Alshehri, Lalana Kagal
arXiv:2506. 21833v2 Announce Type: replace Abstract: Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization are increasingly proposed as memory-efficient, backpropagation-free alternatives for large language model (LLM) fine-tuning, yet their benefits are typically evaluated only against standard backpropagation (BP), omitting memory-efficient variants such as activation checkpointing.
By Kunjal Panchal, Sunav Choudhary, Yuriy Brun, Hui Guan
arXiv:2606. 09079v1 Announce Type: cross Abstract: Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving.
By Yan Wang, Qifan Zhang, Jiachen Yu, Tian Liang, Dongyang Ma, Xiang Hu, Zibo Lin, Chunyang Li, Zhichao Wang, Jia Li, Yujiu Yang, Haitao Mi, Dong Yu