arXiv AI
Jul 9
Predicting LLM Safety Before Release by Simulating Deployment
arXiv:2607. 07184v1 Announce Type: cross Abstract: Pre-deployment safety evaluations aim to inform the downstream risks of releasing a new AI model.
Predicting LLM Safety Before Release by Simulating Deployment
Pre-deployment safety evaluations aim to inform the downstream risks of releasing a new AI model. Yet most evaluations provide limited evidence about how often undesired model behavior will occur in deployment: they generally have insufficient coverage, are unrepresentative, and are generally recognizable as tests.
Related stories
arXiv AI
Aug 7
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
arXiv:2608. 06202v1 Announce Type: cross Abstract: Large language model (LLM) benchmark evaluations are routinely used to support claims about model safety, reliability, and deployment readiness.
arXiv AI
Jul 17
ToolAlignBench: Investigating Alignment Conflicts in Tool-Calling Enabled LLMs
arXiv:2607. 14285v1 Announce Type: cross Abstract: Safety alignment in LLMs aims to align models with human values, but which values take precedence when they conflict?
arXiv AI
Jun 11
Models That Know How Evaluations Are Designed Score Safer
arXiv:2605. 28591v2 Announce Type: replace-cross Abstract: The validity of AI safety evaluations depends on models behaving consistently across controlled and deployment settings.
Hugging Face Trending Papers
Sep 2
Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds
The paper addresses the problem of evaluation awareness in alignment testing, where models can detect they are being evaluated rather than deployed. It introduces two methods: critique refinement, which uses extra inference-time compute to generate and refine action candidates for realism, and DISH, an agent harness that narrows the gap between simulation and real deployment. Experiments show that combining both techniques yields greater realism improvements than either alone, demonstrating that automated approaches can enhance alignment evaluation realism more efficiently than simply extending audit duration.
arXiv AI
Jul 17
Democratizing Agent Deployment Safety: A Structural Monitoring Approach
arXiv:2607. 14570v1 Announce Type: new Abstract: AI software development agents are increasingly capable of modifying infrastructure and security critical systems, creating risks where an agent completes its assigned task while covertly weakening safeguards through actions such as broadening permissions, degrading logging, or introducing persistence mechanisms.
arXiv Machine Learning
Jun 25
ClayBuddy: A Framework, Evaluation, & Mitigation of Coding Agent Failures
arXiv:2606. 19380v3 Announce Type: replace-cross Abstract: Software engineering and deployment are increasingly delegated to AI coding agents.
arXiv AI
Sep 3
Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds
The paper introduces two methods to make alignment evaluations more realistic: critique refinement, which adds inference-time compute to generate and refine candidate actions, and DISH, a deployment-imitating harness that narrows the gap between simulation and real deployment. Experiments on multiple target models show that combining both techniques yields greater realism improvements than using either alone. The study demonstrates that automated approaches can enhance evaluation realism more efficiently than simply extending audit duration.
arXiv AI
Aug 3
Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents
arXiv:2607. 29254v1 Announce Type: new Abstract: AI agents extend large language models (LLMs) with external tools, enabling them to perform complex tasks and translate model outputs into consequential real-world actions.
arXiv AI
Jun 17
An Evaluation of Data Leakage Risks in Tool-Using LLM Agents in Realistic Scenarios
arXiv:2606. 17114v1 Announce Type: cross Abstract: AI agents are increasingly being adopted in enterprise and personal settings with access to emails, databases, documents, and other tools where they can read, update, and disseminate sensitive information.
arXiv AI
Jun 2
Benchmarking AI for low-resource contexts: Thinking beyond leaderboards
arXiv:2605. 28508v2 Announce Type: replace Abstract: Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape usability as much as model quality.
OpenAI Blog
Jun 16
Predicting model behavior before release by simulating deployment
OpenAI introduces Deployment Simulation, a method to predict AI model behavior before deployment using real conversation data to improve safety and evaluation accuracy.