The Flow has not summarised this story yet — read it at arXiv Machine Learning.
arXiv:2609.32807v2 Announce Type: replace Abstract: Clinical LLM evaluation often emphasizes answer accuracy; however, accuracy alone does not test counterfactual consistency or demographic robustnes...
The paper investigates how uncertainty quantification fails in selective prediction systems when clinical data exhibit severe class imbalance, leading to systematic exclusion of minority patients. Through experiments on 56 synthetic scenarios and three neurodegenerative disease cohorts, the authors show that standard uncertainty measures become proxies for class membership rather than true ambiguity, and that combining class‑aware correction (CCR) with threshold‑aware metrics restores proper monotonicity and protects minority patients. The study proposes adopting threshold‑aligned uncertainty measures and class‑stratified evaluation to mitigate this bias and improve safety in clinical AI pipelines.
The paper introduces EDGE, a closed‑form statistical test for assessing the calibration of probabilistic binary classifiers, specifically logistic regression. EDGE uses the same binned predicted‑versus‑observed table as a reliability diagram, projects standardized bin residuals onto a small basis of smooth calibration‑distortion shapes, and yields a null distribution that is a weighted sum of chi‑square variables. The method requires only a single pass over the data and a small eigendecomposition, avoiding refitting, resampling, or tuning, and remains robust in sparse or misspecified settings where other binned tests fail.
arXiv:2608.03854v4 Announce Type: replace Abstract: Quantized large language models can run on consumer hardware, which motivates interest in on-premises processing of sensitive data. The reliability...
arXiv:2610.20417v1 Announce Type: new Abstract: Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures a...
arXiv:2603. 25112v2 Announce Type: replace-cross Abstract: Standard evaluation of LLM confidence relies on calibration metrics (ECE, Brier score) that conflate how much a model knows (Type-1 accuracy) with how well its confidence signal tracks that knowledge (Type-2 metacognitive sensitivity).