arXiv Machine Learning
Jun 9
Decoy-Calibrated Failure Audits for Language Models
arXiv:2606. 09046v1 Announce Type: new Abstract: Useful audits reveal not only how often a model fails, but also where its failures concentrate.
Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models
The paper introduces Janus, a method for validating error patterns in language models by comparing error rates across predefined yes/no properties and using shuffled decoy labels to set significance thresholds. Janus requires that a pattern’s error difference surpasses the decoy-derived threshold and is replicated on held‑out data before reporting. Experiments on a controlled code‑finding task confirm several meaningful error patterns, while on MuSiQue and LongBench v2 Janus reports no confirmed patterns for the tested properties, contrasting with standard shuffling tests that sometimes confirm patterns.
Related stories
arXiv Machine Learning
Aug 18
Certifying Compressed Language Models: An Audit and a Statistical Toolkit
arXiv:2608. 15046v1 Announce Type: new Abstract: A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original.
arXiv AI
Aug 26
Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
The study investigates why small language model agents tend to repeat a tool call that just failed. By recording the failed call and its error message in the transcript, the authors measure a negative corrective gain—agents are more likely to repeat the failed action, with a drop of about 1.03 nats per token. The problem is traced to the harness design rather than the model’s understanding of errors, and the authors show that replacing the verbatim call with a runtime-generated description of the failure can reduce this backfiring effect by 76%.
arXiv AI
Jul 14
Interaction Scaling: Grounding the Third Axis of Test-Time Compute
arXiv:2607. 11598v1 Announce Type: new Abstract: There are two standard ways to spend more compute at test time: let a model reason longer, or sample more attempts and keep one.
arXiv Machine Learning
Sep 17
A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality
The paper introduces a calibrated instrument for rigorously measuring how inference optimizations—such as quantization, early‑exit, and speculative decoding—affect the output quality of large language models. It uses a formally calibrated LLM judge that verifies no systematic bias between statistically equivalent outputs and includes a null condition to ensure measured differences are zero. Applying this method, the authors find that a 4‑bit model is indistinguishable from its 16‑bit counterpart, while 3‑bit quantization and early‑exit techniques incur measurable quality losses that vary by language and task, and that token‑certainty‑based acceptance rules cannot reliably identify impactful errors.
Hugging Face Trending Papers
Jul 13
Interaction Scaling: Grounding the Third Axis of Test-Time Compute
There are two standard ways to spend more compute at test time: let a model reason longer, or sample more attempts and keep one. Both share a hidden limit: they are internal.
Hugging Face Trending Papers
Sep 16
A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality
Large language model optimization is an active research area, spanning quantization of model weights, early-exit methods for skipping layers, and speculative decoding. Each track uses its own quality...
arXiv AI
Sep 25
Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases
The paper introduces inexpensive, scalable methods for evaluating language model behavior across different vendors and releases. By running identical public stimuli on a cross‑vendor panel and analyzing transcripts via exact match, LLM‑coded codebooks, or instrumented environments, the authors can quantify model responses at a cost of a few dollars per model. Applying these tools to four years of releases reveals patterns of convergence, resistance, positional stability, and compliance that vary by generation, lab, and harness.
arXiv AI
Jul 9
Specification Grounding Drives Test Effectiveness for LLM Code
arXiv:2607. 06636v1 Announce Type: cross Abstract: Large language models frequently generate code that appears correct on typical inputs yet fails on edge cases, invalid inputs, and other specification-defined corner conditions.
arXiv AI
1d ago
Hard-Gate Candidacy in a Deployed Validator Suite
The paper evaluates hard‑gate candidacy for validators in a deployed generative‑agent system by measuring how well each validator’s firing separates successful from failed builds. Across 13 validators and thousands of builds, only a few checks show statistically significant separation, while many fail to distinguish or never fire. The study highlights that skipped checks are recorded as passes, limiting detectable failure rates and underscoring the need for clearer evaluation records.
arXiv AI
Aug 20
Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots
The paper introduces EvalCEGAR, a method that automatically evolves a metric for evaluating AI-generated answers by iteratively refining a pool of small Python operators that flag potential defects. By using counterexample-guided abstraction refinement, the system identifies pairs of answers that score identically but differ in correctness, prompting the metric to broaden its scope rather than resample. On benchmark datasets, the evolved 55‑line operator closes a significant portion of the performance gap compared to hand‑written metrics and outperforms a large‑language‑model judge that incurs a cost per candidate.
arXiv AI
Aug 18
Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency
arXiv:2608. 16003v1 Announce Type: new Abstract: Automated checking pipelines increasingly place one language model as the checker and another (or the same one) as the fixer.