arXiv AI
Jun 11
Estimating Tail Risks in Language Model Output Distributions
arXiv:2604. 22167v2 Announce Type: replace-cross Abstract: Language models are increasingly capable and are being rapidly deployed on a population-level scale.
Amplify, Locate, Estimate (ALE) on Rare-Failure Rates of Fine-Tuned LLMs
The paper introduces Amplify, Locate, Estimate (ALE), a technique for efficiently estimating rare failure probabilities in safety‑fine‑tuned large language models. ALE amplifies failure rates by exploiting changes in token log‑probabilities, then uses a score‑function estimator and importance sampling to extrapolate back to the original rare rates. Experiments on 41 targets show ALE’s estimates closely match reference values, achieving accurate predictions for failure rates as low as 10⁻⁵ with far fewer labeled rollouts than direct sampling.
Related stories
arXiv Machine Learning
Sep 22
Rare Event Estimation via Iterative Unalignment
arXiv:2609.24969v1 Announce Type: new Abstract: As agents are deployed with increased autonomy, even extremely rare events along their stochastic output trajectories can occur and prove catastrophic....
arXiv Machine Learning
Aug 4
Caliber: Cross-Architecture Extraction-Cost Control for Score-Returning APIs
arXiv:2608. 01023v1 Announce Type: new Abstract: We present Caliber, an output-perturbation defense against model extraction that formulates noise selection as a calibration problem: how much the defense degrades the supervision signal used to train a surrogate, and the provable per-input query cost of recovering the clean logits.
arXiv Computer Vision
Oct 1
Reliability-Aware Checkpoint Selection for Domain Generalization
arXiv:2609.39934v1 Announce Type: cross Abstract: Checkpoint selection in domain generalization often relies on source-validation accuracy, yet the selected checkpoint need not provide reliable proba...
arXiv AI
Aug 19
Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection
The paper introduces LoRD, a lightweight post‑hoc calibration framework designed to improve confidence reliability in language‑model‑based log anomaly detectors. LoRD learns route‑specific reliability models from latent representations of correctly classified validation samples and uses reconstruction distances to estimate prediction reliability. By selectively recalibrating high‑risk predictions, LoRD reduces overconfident errors while maintaining strong anomaly detection performance across four large‑scale log benchmark datasets.
arXiv Machine Learning
Aug 12
ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions
arXiv:2608. 10621v1 Announce Type: new Abstract: Recent research on Large Language Model (LLM) safety has widely adopted guardrails to identify unsafe LLM outputs.
arXiv AI
Jun 3
ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation
arXiv:2604. 23099v2 Announce Type: replace-cross Abstract: Evaluating generative AI models is increasingly resource-intensive due to slow inference, expensive raters, and a rapidly growing landscape of models and benchmarks.
arXiv Machine Learning
Oct 23
Scaling Laws for Safety Probes
The paper introduces two scaling laws that predict the performance of safety probes—tools used to monitor language models for harmful behavior—based on model size and the number of labeled examples. By creating six large safety datasets and training probes across multiple model families, the authors show that probe error decreases as a power law with model size and further improves with more examples, with the improvement rate increasing for larger models. These laws enable researchers to forecast probe accuracy and decide whether to invest in more data or a larger model to achieve a desired error threshold.
arXiv AI
Aug 28
Invocation-Level Reliability of Tool-Using Agents
The paper investigates the reliability of tool‑using agents, focusing on two failure modes: selecting the wrong tool and constructing incorrect arguments. It introduces a correct‑invocation rate metric to distinguish these errors and evaluates five open‑weight models on multi‑step tasks up to depth 8, finding that by depth 6 about 70% of a model’s clean‑context capability is lost due to earlier mistakes. The study reveals that exact‑match scoring against a fixed gold trajectory forces severity and recovery parameters to extreme values, and proposes a conditional‑on‑state scoring remedy that yields more realistic severity estimates.
arXiv AI
Oct 16
Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence
The paper introduces RADAR, a framework for repairing and auditing confidence scores in large language models (LLMs). It separates confidence repair from deployment decisions and from error auditing, proposing Probe‑Anchored Isotonic Calibration (PAIC) for monotone mapping and a prompt‑clustered kernel test for detecting residual errors. Experiments across checkpoints and benchmarks show that richer predictors can improve confidence estimation, but projecting back to a scalar score can lose gains, and lower prediction risk does not always lead to better review decisions.
arXiv AI
Jun 30
SCARCE: Scalable Cascade Analysis for Rare-event Characterisation via Embeddings
arXiv:2606. 29623v1 Announce Type: new Abstract: Rare events govern the safety profile of modern AI systems, yet their probabilities are extremely difficult to estimate: direct Monte Carlo requires prohibitive sample budgets.
arXiv Machine Learning
Sep 7
Single-Query Black-Box Calibration Auditing via Logit Bias
The paper introduces a method for auditing the calibration of large language models (LLMs) that only exposes a logit_bias parameter. By mathematically manipulating this parameter, the authors can evaluate exact probability thresholds with a single query per sample, enabling a provably consistent estimator of True Calibration Error for binary tasks. This approach offers an efficient framework for auditing black‑box foundation models despite limited access to continuous output probabilities.