arXiv AI By Garud Iyengar, Yu-Shiou Willy Lin, Kaizheng Wang

Model-Free Assessment of Simulator Fidelity via Quantile Curves

Read the original on arXiv AI →

The paper introduces a model‑free method for assessing how well simulators match real‑world systems by constructing confidence sets for latent population parameters across different scenarios. It then uses these sets to estimate a quantile function that provides a distribution‑level risk profile of the simulator, enabling statistical inference, risk measures such as CVaR, and comparisons between simulators. The approach is agnostic to model type and applicable to diverse output spaces, and is demonstrated on four large language models using the WorldValueBench dataset.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Sep 10

Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs

The paper argues that relying solely on zero‑shot task accuracy is insufficient for evaluating quantized large language models (LLMs) because accuracy ignores changes in the full predictive distribution. It proposes a distribution‑sensitive framework that measures fidelity loss by computing statistical distances—such as Jensen‑Shannon Divergence and Total Variation Distance—between the full‑vocabulary output distributions of a full‑precision BF16 reference and its quantized counterparts. Experiments across five foundation architectures and four reasoning benchmarks show that these divergence metrics increase with stronger quantization, revealing distributional drift that top‑1 accuracy fails to capture, and suggest that mixed‑precision Q4_K schemes can offer lower divergence than uniform Q4_0 at comparable memory usage.

By Shahzeb Qamar, Lorenz Sparrenberg, Christian Bauckhage, Baha Rababah, Carson Leung, Murat Kantarcioglu, Cuneyt Gurcan Akcora, Rafet Sifa