arXiv:2609.05881v2 Announce Type: replace-cross Abstract: Developers download quantized large language models (LLMs) from public registries to run them locally. None of the registries we examined doc...
The paper "Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair" investigates why large language model (LLM) code‑repair agents sometimes produce incorrect patches. By analyzing 1,844 model requests on two synthetic Python tasks—PersistQueue and Dijkstra—the authors trace how misconceptions in a model’s hypothesis or implementation plan lead to specific software obligation violations. They find that shifting a misconception from hypothesis to plan increases failure rates by roughly 30% for both tasks, revealing distinct, model‑dependent failure patterns linked to upstream misunderstandings.
The study investigates how language models equipped with tools can still produce unsupported final claims, even when a single tool call could resolve the uncertainty. It defines two metrics—occurrence (how often unsupported claims arise) and conditional repair (how often they are fixed when evidence is provided). Experiments on Qwen3-32B and Gemma 4 show that providing the missing evidence consistently repairs all unsupported claims in the Qwen3-32B setup, while the Gemma 4 model never produced unsupported claims under the tested conditions.
The paper investigates hallucination in large language model–based automated program repair (APR). It defines hallucination as producing patches or intermediate artifacts that are not grounded in available repair evidence, and analyzes it across final patches and intermediate tasks such as triggering test case identification, line coverage prediction, and additional test case generation. Experiments on 832 Defects4J bugs show that only 21.0%–55.9% of patches pass the developer test suite, with 72.7% of sampled repairs exhibiting hallucinations, often due to incorrect causal localization or repair strategies.
arXiv:2610.33949v1 Announce Type: cross Abstract: Pipelines that require structured outputs from hosted language models often specify a Pydantic schema in the prompt, validate the response, and retry...
arXiv:2609.14758v1 Announce Type: cross Abstract: Tool-augmented language models are evaluated on whether they reach the right answer, not on whether they report honestly when a tool fails to supply...
arXiv:2607. 27275v1 Announce Type: new Abstract: Post-training quantization to 4-bit weights is widely reported to be nearly lossless.
The paper introduces a 400‑step ledger task that evaluates language models on three granular components: duplicate recognition, local processing, and counter tracking. Experiments on Qwen3‑8B show that duplicate‑recognition errors increase only late in the episode, while the counter remains flawless over 8,000 steps and local processing stays stable. Adding irrelevant content before each step causes the counter to fail consistently, but an external reminder can restore its accuracy.
The paper reports a preregistered audit of language‑model judges used as measurement instruments, revealing that the assumption that a model’s responses remain stable over time is invalid. Across nearly 53,000 audited requests, repeat rankings and byte‑identical replays fell far below required reliability thresholds, with three identified mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—explaining the discrepancy. The study proposes a three‑level snapshot‑identity framework, eight design rules, and a reporting checklist to prevent such reliability failures in future evaluations.
arXiv:2610.36947v1 Announce Type: cross Abstract: Can additional specialist language-model contexts improve software assurance when deterministic tool evidence is already available? We study this que...
arXiv:2608. 11822v1 Announce Type: cross Abstract: A growing body of work reports that language models represent task-relevant latent structure that they fail to use.
The study investigates how post‑training quantization (8‑bit vs. 4‑bit) affects language‑model agents’ ability to recover from temporary tool failures. Using Llama‑3.1‑8B‑Instruct and Qwen2.5‑7B‑Instruct on twenty deterministic tool‑use tasks across five prompts, the authors find that recovery performance varies with prompt, evaluation target, and scoring policy, sometimes favoring 8‑bit, sometimes 4‑bit, and even reversing direction under different conditions. The results highlight that a single prompt, task set, or scoring rule cannot reliably determine the robustness of quantized agents. whyItMatters":"The findings emphasize that evaluation design choices critically influence conclusions about quantized‑agent robustness, underscoring the need for comprehensive, matched‑task assessments before deployment decisions are made."