arXiv AI

RSD-Poker: Structure-Adaptive and Shift-Robust Risk-Utility Certification for Residual Policies in Imperfect-Information Games

RSD-Poker is a certification framework for residual policies in imperfect‑information games that adapts to game structure and remains robust to shifts in deployment state mixtures. It freezes a bank of residual families and a policy‑visible partition, then assigns each candidate‑group pair weighted upper and lower certificates for risk and utility. The framework guarantees that, with high probability, the selected candidate map meets its declared risk budget and utility bounds across a predeclared uncertainty set of deployment group proportions.

arXiv Machine Learning

Sep 21

Available Guardrails: Certifying Selective Prediction across ML Systems

The paper introduces a method to certify selective prediction in machine learning systems by computing the availability of safety gates through exact-binomial inversion and dynamic programming. It demonstrates that a truth-informed planner can significantly improve mean coverage over naive approaches, and that reallocating error budgets further enhances coverage across diverse applications such as LLM tool‑calling, content moderation, lesion classification, and recommendation. The study highlights the importance of planning and finite‑sample estimation in ensuring reliable, granular deployment of selective predictors.

By Parivesh Priye, Yufeng Wang, Haibin Ling, Michael Chaykowsky

llms safety

arXiv AI

Aug 11

Private Anytime Selective-Risk Certification for Federated Retrieval-Augmented Generation: Guarantees and Empirical Limits

arXiv:2608. 07913v1 Announce Type: cross Abstract: Selective-risk certificates promise that accepted outputs meet a declared error target.

By Sanjeda Akter, Ibne Farabi Shihab, Anuj Sharma

rag nlp safety

Hugging Face Trending Papers

Aug 8

Private Anytime Selective-Risk Certification for Federated Retrieval-Augmented Generation: Guarantees and Empirical Limits

Selective-risk certificates promise that accepted outputs meet a declared error target. We develop Fed-SRC, a score-agnostic certificate for federated, differentially private, adaptively monitored retrieval-augmented generation.

rag nlp safety

arXiv Machine Learning

Oct 15

SafeAdapt-Poker: Counterfactual Risk Calibration for Budgeted Adaptation under Opponent Shift

arXiv:2610.20473v1 Announce Type: cross Abstract: Opponent histories can mislead after behavioral shifts. SafeAdapt-Poker formulates budgeted adaptation under opponent shift as a dual-counterfactual...

By Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Rui Chen, Daren Zha, Jun Xiao

safety

arXiv Machine Learning

Aug 18

Guaranteed Adaptive Modality Acquisition: When the Policy Chooses Its Own Calibration Group

arXiv:2608. 15520v1 Announce Type: new Abstract: A multimodal system may begin inference holding only some of its inputs and may acquire the rest at a cost.

By Melika Baghi

multimodal benchmarks

arXiv Machine Learning

Sep 10

Conditional Validity for Adaptive Modality Acquisition: When the Policy Chooses Its Own Calibration Group

The paper introduces RouteCert, a method for ensuring risk control in multimodal systems that acquire inputs adaptively. It shows that conditional calibration can remain valid even when the acquisition policy determines the calibration group, and provides two finite‑sample constructions: threshold‑free routing with terminal‑pattern calibration and simultaneous validation of policy‑pattern pairs. Experiments on a clinical ECG task and masked multimodal benchmarks demonstrate that RouteCert achieves low disagreement rates and competitive answered fractions while validating each acquisition stage separately.

By Melika Baghi

multimodal benchmarks

arXiv AI

Oct 27

Conformance as a Distribution: Delegation Contracts, Error Budgets, and the Allocation of Evaluation Authority in Use-Based AI Regulation

arXiv:2610.33112v1 Announce Type: new Abstract: Use-based AI regulation assigns obligations by deployment risk but lacks an evidentiary object to carry them. A benchmark score is a property of a mode...

By John deVadoss

llms benchmarks

arXiv AI

Sep 10

Generator-Independent Runtime Assurance under Partial Observation

arXiv:2609.06036v1 Announce Type: new Abstract: Proposal-based controllers---learned policies, language-model planners, and other black-box \emph{generators}---are increasingly deployed behind runtim...

By Guangxi Wan, Yongbo Xie, Yuqi Liu, Qingwei Dong, Qingxin Li, Hongfei Bai, Peng Zeng

safety

arXiv AI

Oct 1

How Much Can Reliability Drift Under a Fixed Confidence Distribution?

arXiv:2609.38917v1 Announce Type: new Abstract: A classifier's conditional accuracy can change while its confidence distribution stays exactly the same. We study the worst-case movement of the reliab...

By Wenhao Liang, Lin Yue, Wei Emma Zhang, Mingyu Guo, Olaf Maennel, Weitong Chen

arXiv AI

Aug 18

Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays

arXiv:2608. 14639v1 Announce Type: cross Abstract: Per-field accept/review with selective risk at most alpha -- accept a field only if the error rate among accepted fields is controlled -- is the trust contract document-extraction systems need, and the natural procedure silently violates it on real documents.

By Bhaskar Gurram

llms

arXiv Machine Learning

Jun 30

When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation

arXiv:2606. 29054v1 Announce Type: new Abstract: Large language models (LLMs) deployed for structured generation (NER, JSON extraction, QA, and classification) lack formal reliability guarantees, and standard heuristic abstention policies miss user-specified risk targets by 7.

By Varun Kotte

llms

arXiv AI

Oct 19

Audit-First VAPO: Risk-Certified Selective Updates under Imperfect Verification

The paper introduces Audit-First VAPO, a method that separates directional admission from magnitude control for selective updates under imperfect verification. It employs an observation-only accept‑appeal‑abstain policy with a finite secondary‑verification budget, freezing the action trace before clean labels are added, and uses finite‑sample bounds to certify harmful risk, coverage, and verifier‑call rate. Experiments on Qwen3.5‑0.8B and GSM8K show that RC‑VAPO achieves high accuracy while maintaining low selected harmful risk and modest verifier cost compared to baseline methods.

By Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Rui Chen, Daren Zha

benchmarks

Related stories