arXiv AI
Counterfactual Fragility Certificates: Exposing High-Confidence Brittleness under Structured Evidence Failure
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 22
FALCON-Discover: Discovering Concentrated False-Confidence Regions for Calibration
arXiv:2607. 18278v1 Announce Type: cross Abstract: Calibration is usually evaluated in aggregate, but the most dangerous failures are often local: predictions that remain highly confident despite being wrong.
arXiv Machine Learning
Jul 22
Signed Evidence Flow: Conflict-Aware and Stability-Calibrated Data Analysis
arXiv:2606. 21875v2 Announce Type: replace-cross Abstract: Modern data analysis usually gives a prediction without showing whether the evidence behind it is clear, conflicting, or stable.
arXiv AI
Aug 13
Claim-Level Reliability Assessment for Efficient Test-Time Reasoning
arXiv:2608. 11994v1 Announce Type: new Abstract: We propose claim-level falsification as a principle for test-time scaling and instantiate it through Claim-Level Reliability Assessment (CLR), a training-free framework that reallocates test-time compute from additional solution sampling to targeted verification.
arXiv Machine Learning
Aug 11
Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk
arXiv:2608. 08126v1 Announce Type: new Abstract: Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters, in tension with supervisory expectations that adverse decisions be explainable.
arXiv AI
Jul 22
Calibrated Selective Fact-Checking via Evidence Chain Evaluation
arXiv:2607. 18240v1 Announce Type: new Abstract: Large language models (LLMs) can achieve strong fact-checking accuracy, yet forced binary decisions conceal a critical reliability problem: systems may issue confident verdicts even when supporting evidence is weak, sparse, or internally inconsistent.
arXiv AI
Aug 24
No Judgment Without a Reason: Counterfactual Receipts for Versioned AI Evaluators
The paper introduces a framework for evaluating AI systems that not only checks final labels but also tracks the reasoning behind them through three core sources—grounds, norms, and authority—forming an eight-cell counterfactual judgment cube. It defines minimal source replacement sets, called judgment receipts, to explain changes in verdicts and provides certification cost bounds for black-box evaluators. The authors present ReasonBench, a benchmark with 19,520 cases, and demonstrate that while high standard accuracy can mask robustness issues, receipt accuracy reveals significant gaps in reasoning consistency across different models.