arXiv AI

Model-Free Assessment of Simulator Fidelity via Quantile Curves

The paper introduces a model‑free method for assessing how well simulators match real‑world systems by constructing confidence sets for latent population parameters across different scenarios. It then uses these sets to estimate a quantile function that provides a distribution‑level risk profile of the simulator, enabling statistical inference, risk measures such as CVaR, and comparisons between simulators. The approach is agnostic to model type and applicable to diverse output spaces, and is demonstrated on four large language models using the WorldValueBench dataset.

arXiv AI
Sep 10

Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs

The paper argues that relying solely on zero‑shot task accuracy is insufficient for evaluating quantized large language models (LLMs) because accuracy ignores changes in the full predictive distribution. It proposes a distribution‑sensitive framework that measures fidelity loss by computing statistical distances—such as Jensen‑Shannon Divergence and Total Variation Distance—between the full‑vocabulary output distributions of a full‑precision BF16 reference and its quantized counterparts. Experiments across five foundation architectures and four reasoning benchmarks show that these divergence metrics increase with stronger quantization, revealing distributional drift that top‑1 accuracy fails to capture, and suggest that mixed‑precision Q4_K schemes can offer lower divergence than uniform Q4_0 at comparable memory usage.

By Shahzeb Qamar, Lorenz Sparrenberg, Christian Bauckhage, Baha Rababah, Carson Leung, Murat Kantarcioglu, Cuneyt Gurcan Akcora, Rafet Sifa
arXiv Machine Learning
Oct 7

A perspective note on likelihood approximation and inference for complex simulation models using a chain of aggregated normalizing flows

The paper proposes a new approach to likelihood approximation for complex simulation models using a chain of aggregated normalizing flows. It describes a sequential estimation strategy where each set of bijective transformations is optimized independently, enabling scalable and controllable simulation routines for large‑scale data analysis. The framework also offers a surrogate model that facilitates sample generation, Bayesian inference, hypothesis testing, and uncertainty quantification.

By Getachew K Befekadu
arXiv Machine Learning
Oct 5

Reliability of Probabilistic Emulation of Physical Systems

The paper compares two main strategies for producing probabilistic forecasts of physical systems: generative models (e.g., diffusion or flow matching) and ensembles of deterministic models trained with the continuous ranked probability score (CRPS). It introduces a framework that evaluates both methods on diverse 2‑D spatiotemporal systems, measuring empirical coverage of predictive intervals, accuracy, and computational efficiency. The study finds that CRPS‑trained ensembles generally provide more reliable uncertainties and faster inference, while generative models trained in ambient space can match coverage but at higher latency. The authors also release AutoCast, a modular framework for both approaches, and AutoSim, a dataset generation tool.

By Sam F. Greenbury, Radka Jersakova, Paolo Conti, Marjan Famili, Christopher Iliffe Sprague, Edwin Brown, Farhan Feroz, Jason D. McEwen