arXiv:2606. 09046v1 Announce Type: new Abstract: Useful audits reveal not only how often a model fails, but also where its failures concentrate.
By Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh
The paper evaluates partial‑prompt completion tests—where a model is given the first part of a benchmark problem and checked for reproducing the rest—on seven open‑weight models using MATH‑500 and recent AIME 2025/2026 problems as true negatives. It finds that memorization flags are highly sensitive to the amount of text left: problems with fewer than eight remaining words are flagged far more often than those with 25 or more, and most ROUGE‑based flags arise from short leftovers. On true negatives, even a high ROUGE‑L threshold flags up to 6.7% of problems at a 60% prefix, and exact‑match flags 8.6% at an 80% prefix, while models distilled from Qwen2.5‑Math show almost no recall of contamination. The authors recommend length‑stratified reporting, limiting prefixes to 60%, using style‑matched post‑cutoff controls, and cross‑family calibration to reduce false positives.
By Abhishek Anand
arXiv:2608. 15046v1 Announce Type: new Abstract: A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original.
By Amogh Singh
The paper investigates how failure taxonomies for large language model (LLM) reasoning can be misleading, focusing on a category called Compositional Collapse. By measuring this category across twelve models and eleven languages on a multilingual math benchmark, the authors find that the reported fraction of failures is highly sensitive to the order of classification tests and that human annotators rarely agree with the automated inspector on this category. They also show that the inspector’s labeling is largely driven by the prompt rather than the model itself, and that the category is over‑represented compared to native speaker judgments.
By Madhu Sudhan Rao Beesetty
The paper investigates whether the tendency of code language models to default to Python when no language is specified is driven by a small, identifiable set of internal components. Using a comprehensive pipeline—including ablation, activation patching, greedy circuit extraction, and significance testing—across 55 model-language configurations, the authors find that this preference is large, grows with model size, and persists after instruction tuning. They also show that the apparent localization of this behavior depends heavily on analysis choices such as component budget and probe design, and that interventions can sometimes alter the model’s output.
By Harsh Kumar, Tanmay Shailesh Joshi, Rahul Maity, Anusa Saha
The study compares single-agent and multi-agent large language model (LLM) setups for reviewing credit assessment memos against specified requirements. In a controlled experiment with 300 synthetic memos, a single agent examined six requirements in one call, while a team of three agents each checked two requirements. Results show the team identified 0.08 to 0.10 more defects on certain models but also produced more false alarms, used 2.2 to 2.4 times more tokens, and had mixed effects on balanced accuracy depending on instruction and model.
By Pei Liu