Tetra introduces a new Leech‑lattice based codebook that reduces the memory footprint of quantized LLMs to about 2.15 bits per weight, enabling efficient 2‑bit quantization without a massive lookup table. The method employs a 64‑state Golay trellis and a shared 16 KiB table, decoding each 24‑weight block with only six table loads and two small lookups. When applied to Qwen3 models (4B, 8B, 14B), Tetra achieves 2.70–2.73 bits per parameter, scoring 63–75 on MMLU and generating 57–114 tokens per second, while maintaining close performance to 4‑bit AWQ and outperforming llama.cpp’s IQ2_XXS on 4B.
The paper introduces DuoRate, a method for allocating precision in Mixture-of-Experts (MoE) models that separates storage and decoding rates. It stores a uniform tier of weights once and then adapts read precision per workload using a Lagrangian rate‑distortion optimization, reducing the store‑read gap. Experiments on Qwen3-30B-A3B and GSM8K show that DuoRate achieves higher accuracy at lower bit‑per‑weight (bpw) rates compared to fixed‑format baselines.
By Mengqi Guo, Zhe Wei, Yuan Yuan, Jiunn Bin Lim, Boyi Pan, Michael Bi Mi
TableTune is a system that automatically measures and corrects draft-length tables for speculative decoding in large language model serving. It profiles decoding performance across batch sizes and draft lengths, generates an optimized table in minutes, and a controller plugin continuously updates the table based on live traffic. Experiments on H100 GPUs show that TableTune consistently achieves at least 94% of the best fixed draft-length throughput, outperforms other speculative control methods, and can recover up to 96% of performance lost by stale tables, delivering significant speedups in real deployments.
By Abhishek Malvankar, Paul Castro, Nick Mitchell, Lionel Villard
arXiv:2608. 08910v1 Announce Type: cross Abstract: PTQTP decomposes LLM weight matrices into two ternary (trit) planes with two free per-group scales.
By Matteo Grella
arXiv:2608. 20210v1 Announce Type: cross Abstract: Small language models are usually built like large ones and then squeezed onto a CPU afterwards.
By Christos Koutsiaris
Coding agents re-send large file reads and tool outputs to a frontier LLM every turn, and this context dominates their token bill. General-purpose prompt compressors are trained on prose and suit code...
Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it.
Large Language Models (LLMs) are increasingly used to optimize code for performance, but this study shows that such optimizations often increase energy consumption rather than reduce it. Across 11 open‑source models and 1,000 benchmark problems, inference energy ranged from 0.79 to 8.68 MJ, and 42 of 44 configurations produced code that used 4% to 886% more energy than human‑written reference solutions. Mixture‑of‑Experts architectures and token‑limit constraints can reduce inference energy, yet performance‑optimized code remains energy‑heavier, indicating that speed and sustainability are not always aligned goals.
By Sadra Emami, Benjamin Weigell, Mohammad Reza Mousavi, Gunel Jahangirova, Hugo Araujo, Bernhard Bauer
The paper investigates why large‑language‑model coding agents rarely request a second chunk of tool output, focusing on the precision‑at‑1 rate ($p_1$) of the gold item appearing first in the first chunk. In a benchmark of 500 software‑engineering tasks, the authors compare six value functions and find that increasing $p_1$ does not systematically improve downstream accuracy; the agent can recover the correct answer from any position within the chunk. Adding file‑metadata signals to a keyword scorer actually reduces $p_1$, while a parameter‑free keyword scorer improves $p_1$ but still fails to boost overall accuracy.
By Tatiana Petrova, Andrei Mazniak, Radu State
arXiv:2610.05685v2 Announce Type: replace
Abstract: Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed m...
By Runguo Li
The paper demonstrates that the number of candidates generated during test-time scaling of large language models does not fully capture the system cost. By comparing different generation schedules (e.g., one batched call versus multiple serial calls) while keeping the total candidate count fixed, the authors show that serial calls consume significantly more GPU energy and latency. The study suggests that reporting candidate count alone is insufficient; evaluations should also include generation schedule and GPU-level metrics.
By Mobina Kashaniyan, Ali Jannesari
Paritok-4B is a 4‑billion‑parameter LoRA compressor designed for coding agents, which extracts and retains key spans of code rather than paraphrasing them. It is intent‑conditioned, selecting lines that are most relevant to the agent’s current task, and achieves high fidelity with 96% of identifiers, paths, and numbers preserved. Trained on 67,074 real OpenHands trajectories and fine‑tuned on Qwen3‑4B, it compresses agent context to about 25.7% of its original size while keeping 86.5% of the uncompressed solve quality across 300 SWE‑bench Lite instances.
By Jiayu Shi, Luzhuo Chen