The paper investigates why large language models often miscount items in bracketed lists. It finds that different instruct models exhibit distinct error patterns—Qwen and Gemma 27B tend to round odd lengths to nearby even numbers, OLMo focuses errors on mid-sized integers, and Llama under‑counts—challenging the idea that a single cause explains all failures. The study also shows that a linear probe can recover the correct count from the residual stream and that fixes effective for one model may not transfer to others.
arXiv:2605. 09239v2 Announce Type: replace-cross Abstract: Large language models fail at counting how many times a word repeats in a list, even though they perform well on far harder reasoning tasks.
arXiv:2607. 09999v1 Announce Type: cross Abstract: We show that post-training quantization can silently alter how large language models reason even when task accuracy is preserved.
arXiv:2608.31068v1 Announce Type: new Abstract: When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of...
The paper investigates whether large language models’ reasoning traces truly contain early, informative signals or merely reflect budget and difficulty confounds. Using a restart‑controlled truncation probe, the authors compare continuation success rates against from‑scratch restart curves across 178 problem‑model pairs, finding that only one case shows prefix‑limited success and that continuing a model’s own prefix generally outperforms restarting. A difficulty‑controlled test and two generation‑free analyses reveal that early internal signals do not carry outcome information beyond a problem‑difficulty baseline, underscoring the need for proper counterfactual controls.
The paper investigates whether large language model (LLM) agents maintain complementary consistency when reporting probabilities for a claim and its negation in opinion simulations. Using three small open models (Qwen3‑4B, Llama‑3.2‑3B, Gemma‑3‑4B), the authors find that paired reports often sum to more than one, even on real fact‑checking claims, and that this inconsistency persists across different simulation settings. Averaging a report with its complement restores consistency but only marginally improves predictive accuracy for one model.
The paper investigates where post‑training quantization (PTQ) harms large language models (LLMs) and how to best allocate a limited precision budget. By causally raising each layer to 8‑bit precision across nine open‑weight models, the authors find that quantization damage is diffuse rather than concentrated in specific task circuits or weight statistics, and that globally refining quantization granularity outperforms selectively protecting the most recoverable layers. They also observe that the residual accuracy loss is budget‑limited and that peak recovery locations correlate with architecture within families but not across families.
The paper introduces a 400‑step ledger task that evaluates language models on three granular components: duplicate recognition, local processing, and counter tracking. Experiments on Qwen3‑8B show that duplicate‑recognition errors increase only late in the episode, while the counter remains flawless over 8,000 steps and local processing stays stable. Adding irrelevant content before each step causes the counter to fail consistently, but an external reminder can restore its accuracy.
arXiv:2610.05533v2 Announce Type: replace-cross Abstract: Harness search keeps a change to the prompts, token budgets or parsers around a frozen model if the change raises a score. Such a gain can co...
arXiv:2608. 08029v1 Announce Type: cross Abstract: Khatri et al.
The paper introduces Janus, a method for validating error patterns in language models by comparing error rates across predefined yes/no properties and using shuffled decoy labels to set significance thresholds. Janus requires that a pattern’s error difference surpasses the decoy-derived threshold and is replicated on held‑out data before reporting. Experiments on a controlled code‑finding task confirm several meaningful error patterns, while on MuSiQue and LongBench v2 Janus reports no confirmed patterns for the tested properties, contrasting with standard shuffling tests that sometimes confirm patterns.
arXiv:2609. 31181v1 Announce Type: new Abstract: Black-box model identification works by scoring a model's response to natural-language prompts.
The paper investigates whether accuracy and order sensitivity in large language model (LLM) multiple‑choice question (MCQ) evaluation move together. Using proprietary and open‑weight models on MMLU and ARC‑Challenge, the authors measure order sensitivity with flip rate and recall standard deviation (RStd) and find that these metrics often diverge from accuracy and from each other. They conclude that accuracy alone is insufficient and recommend reporting both flip rate and RStd alongside accuracy.