Decoy-Calibrated Failure Audits for Language Models
arXiv:2606. 09046v1 Announce Type: new Abstract: Useful audits reveal not only how often a model fails, but also where its failures concentrate.
The paper introduces Janus, a method for validating error patterns in language models by comparing error rates across predefined yes/no properties and using shuffled decoy labels to set significance thresholds. Janus requires that a pattern’s error difference surpasses the decoy-derived threshold and is replicated on held‑out data before reporting. Experiments on a controlled code‑finding task confirm several meaningful error patterns, while on MuSiQue and LongBench v2 Janus reports no confirmed patterns for the tested properties, contrasting with standard shuffling tests that sometimes confirm patterns.
arXiv:2606. 09046v1 Announce Type: new Abstract: Useful audits reveal not only how often a model fails, but also where its failures concentrate.
The paper evaluates partial‑prompt completion tests—where a model is given the first part of a benchmark problem and checked for reproducing the rest—on seven open‑weight models using MATH‑500 and recent AIME 2025/2026 problems as true negatives. It finds that memorization flags are highly sensitive to the amount of text left: problems with fewer than eight remaining words are flagged far more often than those with 25 or more, and most ROUGE‑based flags arise from short leftovers. On true negatives, even a high ROUGE‑L threshold flags up to 6.7% of problems at a 60% prefix, and exact‑match flags 8.6% at an 80% prefix, while models distilled from Qwen2.5‑Math show almost no recall of contamination. The authors recommend length‑stratified reporting, limiting prefixes to 60%, using style‑matched post‑cutoff controls, and cross‑family calibration to reduce false positives.
arXiv:2608. 15046v1 Announce Type: new Abstract: A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original.
The paper investigates how failure taxonomies for large language model (LLM) reasoning can be misleading, focusing on a category called Compositional Collapse. By measuring this category across twelve models and eleven languages on a multilingual math benchmark, the authors find that the reported fraction of failures is highly sensitive to the order of classification tests and that human annotators rarely agree with the automated inspector on this category. They also show that the inspector’s labeling is largely driven by the prompt rather than the model itself, and that the category is over‑represented compared to native speaker judgments.
The paper investigates whether the tendency of code language models to default to Python when no language is specified is driven by a small, identifiable set of internal components. Using a comprehensive pipeline—including ablation, activation patching, greedy circuit extraction, and significance testing—across 55 model-language configurations, the authors find that this preference is large, grows with model size, and persists after instruction tuning. They also show that the apparent localization of this behavior depends heavily on analysis choices such as component budget and probe design, and that interventions can sometimes alter the model’s output.
The study compares single-agent and multi-agent large language model (LLM) setups for reviewing credit assessment memos against specified requirements. In a controlled experiment with 300 synthetic memos, a single agent examined six requirements in one call, while a team of three agents each checked two requirements. Results show the team identified 0.08 to 0.10 more defects on certain models but also produced more false alarms, used 2.2 to 2.4 times more tokens, and had mixed effects on balanced accuracy depending on instruction and model.
The study investigates why small language model agents tend to repeat a tool call that just failed. By recording the failed call and its error message in the transcript, the authors measure a negative corrective gain—agents are more likely to repeat the failed action, with a drop of about 1.03 nats per token. The problem is traced to the harness design rather than the model’s understanding of errors, and the authors show that replacing the verbatim call with a runtime-generated description of the failure can reduce this backfiring effect by 76%.
arXiv:2607. 11598v1 Announce Type: new Abstract: There are two standard ways to spend more compute at test time: let a model reason longer, or sample more attempts and keep one.
The paper introduces a calibrated instrument for rigorously measuring how inference optimizations—such as quantization, early‑exit, and speculative decoding—affect the output quality of large language models. It uses a formally calibrated LLM judge that verifies no systematic bias between statistically equivalent outputs and includes a null condition to ensure measured differences are zero. Applying this method, the authors find that a 4‑bit model is indistinguishable from its 16‑bit counterpart, while 3‑bit quantization and early‑exit techniques incur measurable quality losses that vary by language and task, and that token‑certainty‑based acceptance rules cannot reliably identify impactful errors.
The paper identifies a "specification-oracle gap" in coding‑agent benchmarks, where tests rely on details (constants, module paths, exact messages) that are not stated in the natural‑language task description. The authors formalise this gap, develop a static audit tool that classifies such dependencies across five languages, and demonstrate that many benchmarks (e.g., SWE‑Milestone) contain tasks whose tests depend on unstated information, leading to higher failure rates for agents that do not reproduce upstream implementation details. The study also shows that the prevalence of these gaps varies with benchmark construction and that removing affected tests does not alter leaderboard rankings but shifts scores within repositories, favoring agents that replicate upstream names. whyItMatters":"The findings reveal that benchmark tests can inadvertently measure an agent’s ability to mimic upstream implementation details rather than its true problem‑solving skills, potentially skewing evaluations and comparisons across agents."
The study measures the exact probability that language models assign to each integer from 1 to N when prompted for a random number, eliminating sampling noise by summing over all spellings. Across 18 open‑weight models and 29 decoding settings, instruction tuning consistently makes the distribution less uniform, introducing a round‑number avoidance heuristic and sharpening endpoint avoidance. While tuned models rarely output the extreme values 1 or N and largely avoid multiples of 10, they do not converge on a single favorite integer across models.
The paper evaluates how well typed decision models, which output probability distributions, actually provide reliable guarantees. By comparing open-weight Laya and Jev 1.13 across 190 evaluations, the study finds that point‑estimate thresholds frequently break promised accuracy, while a Beta lower‑bound rule and Learn‑then‑Test approach maintain guarantees more consistently. The work also explores calibration across languages, the impact of noise, data requirements for fine‑tuning, and the limitations of confidence in predicting outcomes or providing faithful explanations.