COMET is a coordinated multi-layer runtime monitoring framework designed for human-supervised AI-enabled safety systems. It employs three monitoring layers—signal integrity, physics consistency, and the AI system’s competence envelope—whose outputs are combined and displayed via a traffic-light interface. Evaluated on a compressed‑air leak dataset, COMET eliminates the AI’s silent failures under covariate shift and most sensor faults, while maintaining a low false‑alarm rate.
Incorporating AI systems into safety systems requires monitoring to avoid silent faults and ensure safe and reliable operation. Single monitors often do not provide actionable information for human su...
arXiv:2608. 06657v1 Announce Type: new Abstract: Modern cyber-physical and AI-assisted systems couple human operators, AI decision modules, and automated controllers in a single control loop, so trustworthiness depends on the whole loop, not any one model.
arXiv:2608. 02786v1 Announce Type: new Abstract: AI systems can fail silently.
arXiv:2608. 14680v1 Announce Type: new Abstract: Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails.
arXiv:2604. 02478v2 Announce Type: replace Abstract: Deep learning models excel at detecting anomaly patterns in normal data.
arXiv:2603. 13343v3 Announce Type: replace-cross Abstract: Predictive maintenance for connected vehicles offers the potential to reduce unexpected breakdowns and improve fleet reliability, but most existing systems rely exclusively on internal diagnostic signals and are validated on simulated or industrial benchmark data.
The paper introduces AOI (Autonomous Operations Intelligence), a multi‑agent framework that treats automated site reliability tasks as a structured trajectory learning problem under security constraints. It combines a trainable diagnostic system using Group Relative Policy Optimization, a read‑write separated execution architecture, and a Failure Trajectory Closed‑Loop Evolver that turns unsuccessful runs into corrective supervision. On the AIOpsLab benchmark, AOI achieves significant performance gains, including a 66.3% best@5 success rate and improved average metrics on unseen fault types.
The paper introduces SPAR, a closed‑loop simulation platform that couples real‑time AUV control software with a higher‑level orchestration layer for fault injection, prompting, and evaluation of large language models (LLMs) in diagnosing and recovering from anomalies. SPAR enables ensemble testing of LLMs, comparing a frontier model with three locally deployable LLMs on a mass‑shift fault scenario across 480 trials, revealing that model choice significantly affects diagnostic accuracy. The study demonstrates that while the frontier model consistently ranks the correct fault mechanism among its top hypotheses, local models succeed mainly when they follow the full diagnostic procedure, and overall diagnosis and operational decisions appear decoupled in this dataset.
arXiv:2608. 02464v1 Announce Type: cross Abstract: LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself.
arXiv:2607. 19321v1 Announce Type: new Abstract: As AI agents begin to automate AI R&D, we need ways to assess whether their outputs are safe to deploy, even when the agents themselves may be untrusted.
The paper introduces SPAR, a closed‑loop simulation platform that couples real‑time AUV control software with a higher‑level orchestration layer to evaluate large language models (LLMs) for fault diagnosis and recovery. It demonstrates that a frontier LLM outperforms locally deployable models in identifying a mass‑shift fault, and shows that successful diagnosis depends on following a complete diagnostic procedure rather than premature conclusions. The study provides an architecture and ensemble evaluation methodology for LLM‑assisted mission management on low‑power autonomous underwater vehicles.
arXiv:2609.06835v1 Announce Type: cross Abstract: Agentic AI systems execute complex tasks through long-horizon workflows of planning, tool use, and multi-agent coordination. Task failures in these s...