arXiv:2607. 09999v1 Announce Type: cross Abstract: We show that post-training quantization can silently alter how large language models reason even when task accuracy is preserved.
By Renuka Oladri, Mohan Vamsi Varadaraju Priya, Jerry Wu
arXiv:2610.20417v1 Announce Type: new
Abstract: Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures a...
By Jiahong Qian
arXiv:2606. 09046v1 Announce Type: new Abstract: Useful audits reveal not only how often a model fails, but also where its failures concentrate.
By Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh
The paper evaluates partial‑prompt completion tests—where a model is given the first part of a benchmark problem and checked for reproducing the rest—on seven open‑weight models using MATH‑500 and recent AIME 2025/2026 problems as true negatives. It finds that memorization flags are highly sensitive to the amount of text left: problems with fewer than eight remaining words are flagged far more often than those with 25 or more, and most ROUGE‑based flags arise from short leftovers. On true negatives, even a high ROUGE‑L threshold flags up to 6.7% of problems at a 60% prefix, and exact‑match flags 8.6% at an 80% prefix, while models distilled from Qwen2.5‑Math show almost no recall of contamination. The authors recommend length‑stratified reporting, limiting prefixes to 60%, using style‑matched post‑cutoff controls, and cross‑family calibration to reduce false positives.
By Abhishek Anand
The paper introduces Janus, a method for validating error patterns in language models by comparing error rates across predefined yes/no properties and using shuffled decoy labels to set significance thresholds. Janus requires that a pattern’s error difference surpasses the decoy-derived threshold and is replicated on held‑out data before reporting. Experiments on a controlled code‑finding task confirm several meaningful error patterns, while on MuSiQue and LongBench v2 Janus reports no confirmed patterns for the tested properties, contrasting with standard shuffling tests that sometimes confirm patterns.
By Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh
arXiv:2608. 08514v1 Announce Type: new Abstract: We independently reproduce two recent methods for making large language model (LLM) reasoning more reliable, and stress-test them across domains and models (RPC across four new task domains with Qwen3-8B, LCF across four 7-8B models).
By Minhan Cho, Jimin Kweon
The study compares single-agent and multi-agent large language model (LLM) setups for reviewing credit assessment memos against specified requirements. In a controlled experiment with 300 synthetic memos, a single agent examined six requirements in one call, while a team of three agents each checked two requirements. Results show the team identified 0.08 to 0.10 more defects on certain models but also produced more false alarms, used 2.2 to 2.4 times more tokens, and had mixed effects on balanced accuracy depending on instruction and model.
By Pei Liu
arXiv:2608.29517v1 Announce Type: new
Abstract: Large language models (LLMs) are increasingly used as essay graders in learning analytics, evaluated almost exclusively with agreement statistics. Educ...
By Veerendra Kumar Sunkavalli
arXiv:2608. 16643v1 Announce Type: cross Abstract: Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation.
By Yifan Zhang, Rahmatollah Beheshti
arXiv:2608. 15046v1 Announce Type: new Abstract: A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original.
By Amogh Singh
arXiv:2610.31679v1 Announce Type: new
Abstract: Large language models are increasingly used as math tutors, raising two questions: can a single LLM detect a student's error, diagnose the underlying m...
By Woojin Lee, Yoonkyu Woo, Jinah Kim, Hyeok-Min Gwon, Jin-Xia Huang
arXiv:2609.35860v1 Announce Type: cross
Abstract: Sampling based consistency is widely used for hallucination detection, yet aggregate performance can conceal systematic differences in which errors a...
By Pranav Darshan, Pranav A, Sravan Karthick T, Minal Moharir, Ivan P. Yamshchikov