The study examined 305 quantized large language models (LLMs) from public registries, testing them on a 15‑task code screen and 164 HumanEval+ tasks. Only seven low‑bit, small‑parameter models failed to solve more than six tasks, and none of the files were corrupted—reproducibility did not guarantee functionality. The authors released audit data and a tool, recommending pre‑publication screening and calibration of low‑bit tags.
arXiv:2609.05881v1 Announce Type: cross Abstract: Developers increasingly run large language models locally by pulling quantized GGUF artifacts from public registries, yet nothing in the distribution...
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
arXiv:2607. 09999v1 Announce Type: cross Abstract: We show that post-training quantization can silently alter how large language models reason even when task accuracy is preserved.
The study investigates how language models equipped with tools can still produce unsupported final claims, even when a single tool call could resolve the uncertainty. It defines two metrics—occurrence (how often unsupported claims arise) and conditional repair (how often they are fixed when evidence is provided). Experiments on Qwen3-32B and Gemma 4 show that providing the missing evidence consistently repairs all unsupported claims in the Qwen3-32B setup, while the Gemma 4 model never produced unsupported claims under the tested conditions.
arXiv:2609.35889v1 Announce Type: cross Abstract: Tool-using language-model agents select and execute third-party artifacts. Different implementations can return the requested output while producing...
The paper investigates whether the hidden activations of large language models (LLMs) contain signals about the vulnerability of C/C++ code when the code is provided as context. By extracting prefill token activations from four LLMs and training small MLP probes, the authors achieve an average F1 score of 41.7% across four benchmarks, with the best probe matching state‑of‑the‑art fine‑tuned classifiers on the Devign dataset. The results suggest that a coding LLM’s internal representation can inform vulnerability detection, opening the door to lightweight, model‑native screening methods.
The paper investigates where post‑training quantization (PTQ) harms large language models (LLMs) and how to best allocate a limited precision budget. By causally raising each layer to 8‑bit precision across nine open‑weight models, the authors find that quantization damage is diffuse rather than concentrated in specific task circuits or weight statistics, and that globally refining quantization granularity outperforms selectively protecting the most recoverable layers. They also observe that the residual accuracy loss is budget‑limited and that peak recovery locations correlate with architecture within families but not across families.
arXiv:2607. 28545v2 Announce Type: replace-cross Abstract: Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began.
arXiv:2610.19498v1 Announce Type: cross Abstract: Model signing authenticates the full-precision weights W an LLM is released with, yet users run a quantized artifact Q whose producing transformation...
The paper investigates whether large language models (LLMs) can automatically verify no-code bug fixes—solutions that instruct users to change settings, upgrade versions, or alter workflows—by executing them in a real browser environment. Using 322 fixes from a benchmark, the study tests three executor agents (OpenCUA‑72B, Claude Sonnet 5, and Meta’s Muse Glimmer) and finds that only 17.6% of issues could be set up and passed sanity checks, with resolution rates ranging from 14.6% to 49.7% depending on the executor. Even the best configuration resolved fewer than half of the fixes, and executor choice significantly affected outcomes, highlighting the need for execution-based verification before such fixes reach users.
arXiv:2608. 04719v1 Announce Type: new Abstract: Agent evaluations tell us that a model picked the wrong tool, but rarely why.
The paper introduces an action‑class diagnostic framework for multi‑turn tool‑calling in large language model agents, breaking failures into action‑class miscalibration and action‑execution failure across a four‑class action space (TOOL_CALL, ASK, REFUSE, CONFIRM). It defines a self‑revealing upper bound (Acc GAR) to expose state‑grader masking of miscalibration and shows that miscalibration is a significant, previously hidden failure mode, especially for heavily tool‑trained families. The study demonstrates that calibration can be reshaped by context‑only perturbations, but the effects vary widely across models and perturbation mechanisms, underscoring the need for diagnostics beyond aggregate accuracy.