The Flow has not summarised this story yet — read it at arXiv Machine Learning.
arXiv:2608. 02464v1 Announce Type: cross Abstract: LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself.
The paper introduces a claim‑safe protocol for evaluating closed‑loop AI systems, consisting of three actions: Refuse, Decompose, and Refresh. It demonstrates the protocol in a simulator with 24 policy components and 1,440 held‑out cases, showing that abstention and stable false admission rates are low while providing detailed statistical diagnostics. The approach emphasizes that evaluation results should be tied to observable support and statistical calibration rather than a single PASS/FAIL label.
arXiv:2608.21251v1 Announce Type: new Abstract: Operational telemetry can be jointly anomalous while every individual stream stays inside its familiar range. TRACE-C is an auditable strictly-prior ra...
LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are measurements produced by parsers, beh...
arXiv:2606. 15474v1 Announce Type: new Abstract: Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down.
The study evaluates a convolutional neural network (CNN) technosignature detector trained on Breakthrough Listen data by injecting 15,000 synthetic signals of varied morphologies and amplitudes into real Green Bank Observatory backgrounds. Results show that the detector’s recall decreases with increasing signal brightness and fails to detect certain signal morphologies, while some interference patterns produce false positive rates comparable to its baseline recall. The analysis reveals that the detector only processes ON panels, lacking the OFF panel information needed to distinguish genuine extraterrestrial signals from terrestrial interference, which explains both the false alarms and missed detections.