← Back to all news
arXiv Machine Learning By Jerry Kaplan

A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality

Read the original on arXiv Machine Learning →

The paper introduces a calibrated instrument for rigorously measuring how inference optimizations—such as quantization, early‑exit, and speculative decoding—affect the output quality of large language models. It uses a formally calibrated LLM judge that verifies no systematic bias between statistically equivalent outputs and includes a null condition to ensure measured differences are zero. Applying this method, the authors find that a 4‑bit model is indistinguishable from its 16‑bit counterpart, while 3‑bit quantization and early‑exit techniques incur measurable quality losses that vary by language and task, and that token‑certainty‑based acceptance rules cannot reliably identify impactful errors.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Sep 1

When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs

The paper argues that traditional global calibration metrics, such as Expected Calibration Error and Brier Score, are confounded by differences in model accuracy when comparing large language models. It introduces ACE, an accuracy‑controlled evaluation framework that offers Instance‑Aligned, Distribution‑Aligned, and Candidate‑Aligned views to provide fairer cross‑model comparisons. Experiments across various benchmarks reveal that many reported calibration advantages disappear after accuracy control and that model rankings often reverse, indicating that raw global metrics are unreliable for cross‑model calibration assessment.

By Zhichao Yang, Caiqi Zhang, Ruihan Yang, Chengzu Li, Nigel Collier, Deqing Yang
llmsbenchmarks
More like this →
arXiv AI
Oct 26

Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation

The paper argues that relying solely on overall accuracy or F1 scores to validate automatic scorers in LLM evaluation can conceal prompt‑specific errors that distort comparative results. By auditing six models on 180 invented‑entity questions across six prompts, the authors show that a scorer’s overall F1 of .916 masks substantial variation—up to 33.3%—in error rates depending on the prompt. In a second audit, differences in measured prompt effects dropped dramatically when multiple raters were used, highlighting the need to validate the exact comparison rather than the scorer alone.

By Ancil Crayton
llms
More like this →