arXiv Machine Learning

Choosing Quantized Code LLMs by File Size and Energy per Solved Task

Hugging Face Trending Papers
Sep 28

Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

Tetra introduces a new Leech‑lattice based codebook that reduces the memory footprint of quantized LLMs to about 2.15 bits per weight, enabling efficient 2‑bit quantization without a massive lookup table. The method employs a 64‑state Golay trellis and a shared 16 KiB table, decoding each 24‑weight block with only six table loads and two small lookups. When applied to Qwen3 models (4B, 8B, 14B), Tetra achieves 2.70–2.73 bits per parameter, scoring 63–75 on MMLU and generating 57–114 tokens per second, while maintaining close performance to 4‑bit AWQ and outperforming llama.cpp’s IQ2_XXS on 4B.

arXiv Machine Learning
Oct 12

DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving

The paper introduces DuoRate, a method for allocating precision in Mixture-of-Experts (MoE) models that separates storage and decoding rates. It stores a uniform tier of weights once and then adapts read precision per workload using a Lagrangian rate‑distortion optimization, reducing the store‑read gap. Experiments on Qwen3-30B-A3B and GSM8K show that DuoRate achieves higher accuracy at lower bit‑per‑weight (bpw) rates compared to fixed‑format baselines.

By Mengqi Guo, Zhe Wei, Yuan Yuan, Jiunn Bin Lim, Boyi Pan, Michael Bi Mi
arXiv AI
Oct 31

TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding

TableTune is a system that automatically measures and corrects draft-length tables for speculative decoding in large language model serving. It profiles decoding performance across batch sizes and draft lengths, generates an optimized table in minutes, and a controller plugin continuously updates the table based on live traffic. Experiments on H100 GPUs show that TableTune consistently achieves at least 94% of the best fixed draft-length throughput, outperforms other speculative control methods, and can recover up to 96% of performance lost by stale tables, delivering significant speedups in real deployments.

By Abhishek Malvankar, Paul Castro, Nick Mitchell, Lionel Villard
arXiv AI
Oct 26

Green Code or Costly Code? Evaluating the Energy Costs of Iterative LLM-Based Code Optimizations

Large Language Models (LLMs) are increasingly used to optimize code for performance, but this study shows that such optimizations often increase energy consumption rather than reduce it. Across 11 open‑source models and 1,000 benchmark problems, inference energy ranged from 0.79 to 8.68 MJ, and 42 of 44 configurations produced code that used 4% to 886% more energy than human‑written reference solutions. Mixture‑of‑Experts architectures and token‑limit constraints can reduce inference energy, yet performance‑optimized code remains energy‑heavier, indicating that speed and sustainability are not always aligned goals.

By Sadra Emami, Benjamin Weigell, Mohammad Reza Mousavi, Gunel Jahangirova, Hugo Araujo, Bernhard Bauer
arXiv Computation and Language
Aug 28

Agents Don't Paginate: First-Chunk Selection for LLM Tool Responses

The paper investigates why large‑language‑model coding agents rarely request a second chunk of tool output, focusing on the precision‑at‑1 rate ($p_1$) of the gold item appearing first in the first chunk. In a benchmark of 500 software‑engineering tasks, the authors compare six value functions and find that increasing $p_1$ does not systematically improve downstream accuracy; the agent can recover the correct answer from any position within the chunk. Adding file‑metadata signals to a keyword scorer actually reduces $p_1$, while a parameter‑free keyword scorer improves $p_1$ but still fails to boost overall accuracy.

By Tatiana Petrova, Andrei Mazniak, Radu State
arXiv Machine Learning
Sep 18

Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

The paper demonstrates that the number of candidates generated during test-time scaling of large language models does not fully capture the system cost. By comparing different generation schedules (e.g., one batched call versus multiple serial calls) while keeping the total candidate count fixed, the authors show that serial calls consume significantly more GPU energy and latency. The study suggests that reporting candidate count alone is insufficient; evaluations should also include generation schedule and GPU-level metrics.

By Mobina Kashaniyan, Ali Jannesari
arXiv AI
Aug 26

Paritok-4B: Intent-Conditioned Context Compression for Coding Agents

Paritok-4B is a 4‑billion‑parameter LoRA compressor designed for coding agents, which extracts and retains key spans of code rather than paraphrasing them. It is intent‑conditioned, selecting lines that are most relevant to the agent’s current task, and achieves high fidelity with 96% of identifiers, paths, and numbers preserved. Trained on 67,074 real OpenHands trajectories and fine‑tuned on Qwen3‑4B, it compresses agent context to about 25.7% of its original size while keeping 86.5% of the uncompressed solve quality across 300 SWE‑bench Lite instances.

By Jiayu Shi, Luzhuo Chen