arXiv AI

When Can We Trust the Matching Principle? Robust Deployment Geometry Under Finite-Sample and Model Uncertainty

arXiv Machine Learning
Sep 11

Conformal Calibration Transfer

Conformal Calibration Transfer addresses the challenge of applying conformal prediction when labeled calibration data is only available in a source space, while predictions are needed in a target space linked via unlabeled paired observations. The proposed Transported Conformal Calibration (TCC) method transports source calibration into the target domain and then corrects residual mismatches using only unlabeled target inputs, with two variants: TCC‑KS, which conservatively adjusts calibration based on a label‑free uncertainty surrogate, and weighted‑TCC, which reweights transported calibration for efficiency when weights are stable. Finite‑sample target‑domain coverage guarantees are provided, and experiments on CIFAR‑100‑C, Tiny‑ImageNet‑C, and SEN12MS demonstrate reliable coverage transfer without labeled target data, along with label‑free diagnostics to signal when correction is required.

By Achref Doula
arXiv Machine Learning
Sep 24

CS-WCP: Robust Conformal Sets for LLM-Judge Traffic Shifts with Uncertain Group Proportions

CS-WCP introduces confidence‑set weighted conformal prediction to provide robust prediction sets for large‑language‑model judges when deployment traffic shifts the prevalence of task or policy groups. By constructing simultaneous exact intervals for source and target group masses and taking the union over all compatible ratio vectors, CS‑WCP achieves high coverage (mean 0.973) with few failures across 336 constructed traffic shifts, outperforming standard source conformal prediction. The method offers an auditable coverage safeguard under uncertain mixture weights, focusing on conservative tail protection rather than tighter set sizes.

By Ibne Farabi Shihab, Fariya Afrin
arXiv AI
Sep 1

Decomposing Wrong-Consensus Agreement in LLM Self-Consistency

The paper investigates the nature of agreement among repeated samples of large language models (LLMs), showing that strong agreement can arise even for incorrect answers. It introduces a pluralistic agreement index, Gamma, which is decomposed into a mechanical component driven solely by per‑case answer preferences and a residual component that captures preference‑unexplained agreement. Experiments on GPT‑4.1 and several open‑weight models demonstrate that mechanical agreement dominates in many settings, while the residual varies with benchmark type and sampling protocol.

By Lizhuo Zhang, Mengmeng Tang, Chenfeng Long, Xiaoyong Tang, Xiang Luo
arXiv Machine Learning
Aug 26

Replicable Conformal Prediction

arXiv:2608.23638v1 Announce Type: cross Abstract: Two analysts who calibrate the same predictive model on independent samples will deploy different prediction sets every time, because the calibration...

By Marios Papamichalis, Regina Ruane, Theofanis Papamichalis
arXiv Computer Vision
Sep 11

A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction Models

The paper audits the confidence outputs of seven feed‑forward 3D reconstruction backbones across 13 datasets, evaluating four properties: error ranking, average error‑to‑uncertainty ratio, slope of this ratio, and coverage of the implied error distribution. While confidence ranks errors well, the decoded uncertainty is consistently too small—off by at least 2.4× on median cases—and worsens with higher confidence. A post‑hoc power‑law fit per backbone‑dataset pair improves all four metrics at the dataset level, reducing the median error by 1.35×, but fails to correct coverage for many held‑out scenes, indicating the models lack the correct error scale and distribution shape.

By Nanxing Nick Deng, Qing Cheng, Niclas Zeller, Daniel Cremers