arXiv AI
Jun 11
Estimating Tail Risks in Language Model Output Distributions
arXiv:2604. 22167v2 Announce Type: replace-cross Abstract: Language models are increasingly capable and are being rapidly deployed on a population-level scale.
Quantifying Behavioral Tails in Black-Box Language Models
The paper introduces RareTrap, a framework that estimates the probability of severe behaviors in black‑box large language models. RareTrap constructs a geometry‑aware mapping from a low‑dimensional latent space into token‑embedding space using a surrogate LLM, creating an explicit and reproducible distribution over input prompts. By applying a response‑level performance function and sequential rare‑event simulation, RareTrap concentrates evaluations on increasingly severe behaviors while preserving probability, enabling estimation of such behaviors with as few as 200 evaluations across multiple open‑weight and frontier models.
Related stories
arXiv AI
Jul 22
Estimating Rare Events in Language Models with Proper Evaluation
arXiv:2607. 18454v1 Announce Type: cross Abstract: Quantifying the risk of rare failures in language models, such as those triggered by adversarial distribution shifts or very large-scale deployments, requires estimating probabilities far too small for random sampling.
arXiv AI
2d ago
OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models
The paper introduces OTROPE, a likelihood‑free method for off‑policy evaluation of large language models (LLMs) that uses optimal transport to align labeled samples from a behavior model with unlabeled samples from a target model in a semantic space. OTROPE corrects human‑labeled residuals with proxy predictors, achieving a doubly robust evaluation without requiring behavior‑policy modeling or density‑ratio estimation. The authors provide theoretical guarantees for consistency and convergence, and demonstrate through synthetic and real LLM tasks that OTROPE outperforms existing baselines and can elevate weaker evaluators to match or exceed stronger ones.
arXiv Machine Learning
Jul 7
How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation
arXiv:2605. 06605v2 Announce Type: replace Abstract: Evaluating and predicting the performance of large language models (LLMs) in multi-turn conversational settings is critical yet computationally expensive; key events -- e.
arXiv Computation and Language
1d ago
Sequential Bayesian Evaluation of Large Language Model Behavior
arXiv:2511.10661v2 Announce Type: replace Abstract: It is increasingly important to evaluate the characteristics of systems based on large language models (LLMs). Evaluations in this context often re...
arXiv Machine Learning
Sep 22
Rare Event Estimation via Iterative Unalignment
arXiv:2609.24969v1 Announce Type: new Abstract: As agents are deployed with increased autonomy, even extremely rare events along their stochastic output trajectories can occur and prove catastrophic....
arXiv AI
Jul 7
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
arXiv:2607. 03882v1 Announce Type: cross Abstract: LLMs are increasingly deployed as post-hoc explainers of AI-generated outputs, yet it remains unclear whether they can reliably communicate probabilistic information in natural language.
arXiv AI
Jul 3
Safety Targeted Embedding Exploit via Refinement
arXiv:2607. 01859v1 Announce Type: new Abstract: Safety training for large language models (LLMs) is conducted predominantly in English, leaving uncertain how well safety mechanisms generalize to low-resource languages and mixed-language code-switching.
Hugging Face Trending Papers
Jul 2
Safety Targeted Embedding Exploit via Refinement
Safety training for large language models (LLMs) is conducted predominantly in English, leaving uncertain how well safety mechanisms generalize to low-resource languages and mixed-language code-switching. We show that this creates an epistemic gap in which models confidently generate harmful responses for inputs that fall outside the distribution of their safety training.
arXiv AI
Sep 23
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
Pinocchio is an external calibrator that provides fast uncertainty estimates for black-box language models. It predicts the correctness of responses from seven trained LLMs with an AUROC of 0.862 and can transfer zero‑shot to thirteen unseen models from eight organizations. The method requires only a single forward pass and no access to the target model’s internal states, and a lightweight 0.8B checkpoint achieves comparable performance.
arXiv Machine Learning
Aug 4
The Illusion of Stochasticity in LLMs
arXiv:2604. 06543v2 Announce Type: replace-cross Abstract: In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents.
arXiv Machine Learning
Sep 22
Logits are All We Need to Adapt Closed Models
arXiv:2502.06806v5 Announce Type: replace Abstract: Many commercial Large Language Models (LLMs) are often closed-source, limiting developers to prompt tuning for aligning content generation with spe...