arXiv:2608. 02464v1 Announce Type: cross Abstract: LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself.
The paper introduces a claim‑safe protocol for evaluating closed‑loop AI systems, consisting of three actions: Refuse, Decompose, and Refresh. It demonstrates the protocol in a simulator with 24 policy components and 1,440 held‑out cases, showing that abstention and stable false admission rates are low while providing detailed statistical diagnostics. The approach emphasizes that evaluation results should be tied to observable support and statistical calibration rather than a single PASS/FAIL label.
arXiv:2608.21251v1 Announce Type: new Abstract: Operational telemetry can be jointly anomalous while every individual stream stays inside its familiar range. TRACE-C is an auditable strictly-prior ra...
LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are measurements produced by parsers, beh...
arXiv:2606. 15474v1 Announce Type: new Abstract: Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down.
The study evaluates a convolutional neural network (CNN) technosignature detector trained on Breakthrough Listen data by injecting 15,000 synthetic signals of varied morphologies and amplitudes into real Green Bank Observatory backgrounds. Results show that the detector’s recall decreases with increasing signal brightness and fails to detect certain signal morphologies, while some interference patterns produce false positive rates comparable to its baseline recall. The analysis reveals that the detector only processes ON panels, lacking the OFF panel information needed to distinguish genuine extraterrestrial signals from terrestrial interference, which explains both the false alarms and missed detections.
The paper reports a preregistered audit of language‑model judges used as measurement instruments, revealing that the assumption that a model’s responses remain stable over time is invalid. Across nearly 53,000 audited requests, repeat rankings and byte‑identical replays fell far below required reliability thresholds, with three identified mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—explaining the discrepancy. The study proposes a three‑level snapshot‑identity framework, eight design rules, and a reporting checklist to prevent such reliability failures in future evaluations.
arXiv:2608.27808v1 Announce Type: cross Abstract: Self-report is the cheapest oversight channel a deployer has, and on capable computer-use agents (CUAs) it fails precisely where oversight matters. O...
PROVIA is a system for online mistake detection in egocentric videos that tracks procedure state by maintaining a factual state and a learned summary of performed steps, including mistakes. It uses Bayesian state merging to create an automaton from correct demonstrations and applies a sequential test to convert per‑frame mistake probabilities into alarms. Evaluated on datasets such as CaptainCook4D, IndustReal, HoloAssist, and IMPACT‑ego, PROVIA outperforms baseline methods while operating at 58–70 fps.
arXiv:2608. 05685v1 Announce Type: new Abstract: Most public benchmarks for machine-condition monitoring come from test rigs, where faults are induced on purpose and every event is known.
arXiv:2608. 07914v1 Announce Type: new Abstract: Behavioral contamination detectors can return "no evidence" either because a benchmark is clean or because the audit has little power.
The paper introduces an open generator for telecom alarm storms that produces exact root causes and propagation paths for every alarm on real backbone topologies, with each emitted field tied to standardized terms from NORIA‑O, RFC 8632, RFC 8345, or TMF642. Unlike previous benchmarks, the generator does not rely on confidential operator logs and includes fixed configurations, splits, a scorer, and nine baselines. A simple rule—ranking the device that raised the earliest critical alarm—outperforms more complex methods on both synthetic and real operator data, highlighting a key property of alarm streams.