arXiv:2609.06473v1 Announce Type: new Abstract: Inference-time activation steering enables behavioral control of large language models without parameter modification, while post-training quantization...
The study investigates how post‑training quantization (PTQ) affects proactive interference (PI) in large language models. Using bitsandbytes, the authors compare FP16, INT8, and INT4/NF4 precision across three instruction‑tuned models and find that INT4 quantization markedly degrades accuracy under high interference, with INT8 also incurring a smaller penalty in two of the three models. The degradation is linked to increased same‑key intrusion errors and originates in the quantized transformer backbone rather than the output layer.
Standard decoding rules for autoregressive language models promote diversity by rescaling the full next-token distribution or truncating its low-probability tail. These strategies overlook a common re...
The paper investigates where post‑training quantization (PTQ) harms large language models (LLMs) and how to best allocate a limited precision budget. By causally raising each layer to 8‑bit precision across nine open‑weight models, the authors find that quantization damage is diffuse rather than concentrated in specific task circuits or weight statistics, and that globally refining quantization granularity outperforms selectively protecting the most recoverable layers. They also observe that the residual accuracy loss is budget‑limited and that peak recovery locations correlate with architecture within families but not across families.
XTC (Exclude Top Choices) is a lightweight, head‑aware decoding operator that improves diversity in autoregressive language models by removing overly probable tokens that dominate the next‑token distribution. It works by identifying tokens above a plausibility threshold, probabilistically excluding the dominant choices, and renormalizing the remaining distribution. Across 60 experiments on models such as Gemma 3 and DeepSeek R1, XTC boosts Distinct‑2 scores by 11–15 % and cuts repeat trigrams by 27–47 %, while a Mechanical Turk study shows a 62.3 % preference for XTC‑generated text without loss of fluency.
arXiv:2608.28809v1 Announce Type: new Abstract: Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$, a...
arXiv:2609.18005v1 Announce Type: cross Abstract: Large language model optimization is an active research area, spanning quantization of model weights, early-exit methods for skipping layers, and spe...
arXiv:2608.28911v1 Announce Type: new Abstract: The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length....
Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is assumed harmless and its safety is rarely re-checked.
arXiv:2607. 23976v1 Announce Type: cross Abstract: Appending a two-word confirmation tag to a decision question -- "Is X the better choice?
arXiv:2608. 06564v2 Announce Type: replace Abstract: Quantization is how large language models are actually deployed, and below four bits it hurts.
Accuracy changes after language-model self-revision are usually interpreted as changes in reasoning. We show this can fail at the answer-extraction boundary, and test the failure causally rather than only observationally.