arXiv Machine Learning By Rohit Patel, Susil Kumar Mohanty, Jeenal Chaudhary

TriFleetRCA: On-Premise LLM Root Cause Analysis for Kubernetes

Read the original on arXiv Machine Learning →

TriFleetRCA is an on‑premise pipeline that performs root‑cause analysis for Kubernetes using a single GPU. It gathers evidence at pod, namespace, or cluster scope, deduplicates and ranks it with BM25, filters runbooks through an ingest guard, and returns a root cause with supporting evidence lines. In a live cluster with four injected faults, the system achieved hit rates of 0.85–0.95 across scopes, improved accuracy with deduplication, and demonstrated robust defense against poisoned runbooks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Sep 18

Refuse, Decompose, Refresh: A Claim-Safe Protocol for Closed-Loop AI Evaluation

The paper introduces a claim‑safe protocol for evaluating closed‑loop AI systems, consisting of three actions: Refuse, Decompose, and Refresh. It demonstrates the protocol in a simulator with 24 policy components and 1,440 held‑out cases, showing that abstention and stable false admission rates are low while providing detailed statistical diagnostics. The approach emphasizes that evaluation results should be tied to observable support and statistical calibration rather than a single PASS/FAIL label.

By Peiying Zhu, Sidi Chang
arXiv AI
Sep 4

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

The paper reports a preregistered audit of language‑model judges used as measurement instruments, revealing that the assumption that a model’s responses remain stable over time is invalid. Across nearly 53,000 audited requests, repeat rankings and byte‑identical replays fell far below required reliability thresholds, with three identified mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—explaining the discrepancy. The study proposes a three‑level snapshot‑identity framework, eight design rules, and a reporting checklist to prevent such reliability failures in future evaluations.

By Haoyaun Zhu, Jie Zhang