arXiv AI By Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao

Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents

Read the original on arXiv AI →

The paper presents a three‑phase comparative study of three evaluation frameworks—trace‑based LLM judging, text‑only LLM judging with red‑teaming, and deterministic heuristics—applied to two analytics agents. Phase 1 shows high stability (97% of configurations) across agents, Phase 2 finds the trace‑based framework delivers the best precision‑recall (F1 = 0.90) while text‑only generates many false alarms, and Phase 3 uncovers 24 novel failures not found by functional testing or security red‑teaming. The authors conclude that the type of information used by the harness dictates evaluation quality, functional and security tests are complementary, and LLM‑based harnesses detect subtle errors missed by humans, recommending a dual‑framework approach.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 9

From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage

The paper introduces ALERT‑BENCH, a benchmark that simulates enterprise security telemetry and evaluates how large‑language‑model agents triage alerts. Five common reasoning strategies—single‑pass tool use, iterative retrieval, sampled investigations, self‑review, and explicit verification—were tested on 1,247 alerts from a multi‑stage attack scenario, each missing at least 40.4% of attack‑related alerts. Building on these findings, the authors propose AIDA, a multi‑agent framework that enforces explicit decision proposals, independent challenges, and stronger evidence requirements, achieving an F1 score of 0.958 and reducing false‑negative alerts from 40.4% to 3.1% while escalating 18.4% to analysts.

By Saimon Amanuel Tsegai (Daphne), Alex Kantchelian (Daphne), Danfeng (Daphne), Yao, Peng Gao
arXiv AI
Oct 19

AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems

AuditKIT is a source‑available evaluation library that unifies benchmark accuracy, judge‑based assessment, runtime performance, tool‑use behavior, and retrieval quality into a single result representation. It offers direction‑aware metric contracts, configuration fingerprints, and extensible metrics and backends, while providing turn‑aware tool‑call matching and distinguishing verified task outcomes from trajectory‑level diagnostics. Seven case studies demonstrate how answer extraction, runtime behavior, and metric selection can alter evaluation outcomes across model comparison, serving runtimes, retrieval, tool calling, and interoperability.

By Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth
arXiv AI
Sep 24

CART: Closed-Loop Adaptive Red Teaming for Large Language Models

CART (Closed‑Loop Adaptive Red Teaming) is a framework that iteratively uses results from red‑teaming tests to guide subsequent probes, thereby expanding risk coverage and maintaining diversity. It separates the roles of Challenger (test generator), Target (model or agent under test), and Judge (result evaluator), enabling independent study of each component. Across multiple evaluation families, CART uncovers more failures and higher risk than static prompt replay, demonstrating that adaptive, continuous testing reveals weaknesses that fixed‑prompt methods miss.

By Dongdong Zhang, Tengchao Lv, Yilin Jia, Yuzhong Zhao, Yupan Huang, Wenshan Wu, Xiangyang Zhou, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Furu Wei