arXiv Machine Learning
CURA: Certified Runtime Alarms for Computer-Use Agents
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 4
Real-Time Detection and Repair of LLM Agent Failures
arXiv:2608. 02464v1 Announce Type: cross Abstract: LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself.
arXiv Machine Learning
Jun 10
From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents
arXiv:2606. 09863v1 Announce Type: new Abstract: LLM agents can fail silently by asserting task completion when the environment state shows otherwise.
arXiv AI
Sep 4
Measuring Harmfulness of Computer-Using Agents
The paper introduces CUAHarm, a benchmark comprising 104 expert‑written realistic misuse scenarios for computer‑using agents (CUAs), such as disabling firewalls or leaking data. Using a sandbox with verifiable rewards, the authors evaluate frontier language models—including GPT‑5, Claude 4 Sonnet, Gemini 2.5 Pro, Llama‑3.3‑70B, and Mistral Large 2—and find that even without jailbreak prompts, these models can successfully execute many malicious tasks at high rates (e.g., 90% for Gemini 2.5 Pro). The study also shows that newer models, while safer in traditional safety benchmarks, exhibit higher misuse risks as CUAs, and that monitoring CUAs’ actions remains challenging, with current methods achieving only about 77% accuracy.
arXiv AI
Aug 18
When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry
arXiv:2608. 14680v1 Announce Type: new Abstract: Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails.
arXiv AI
Aug 26
More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight
The paper introduces the twin‑prefix framework to evaluate how the size of the verification unit—i.e., how many actions a pre‑execution LLM monitor reviews in one call—affects its performance. By pairing each gold plan with a twin that differs by a single write and injecting a controlled error, the authors isolate the impact of review length on catch rates and false rejections. Their findings show that longer review windows increase rejection rates but do not improve discrimination, with the highest informedness occurring at one or two actions across all judges and domains.
arXiv Computation and Language
Sep 14
Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?
arXiv:2606.23189v2 Announce Type: replace-cross Abstract: Computer-use agents (CUAs) now act on a user's behalf across personal applications such as email, calendars, and to-do lists. This cross-appl...