The paper introduces ALERT‑BENCH, a benchmark that simulates enterprise security telemetry and evaluates how large‑language‑model agents triage alerts. Five common reasoning strategies—single‑pass tool use, iterative retrieval, sampled investigations, self‑review, and explicit verification—were tested on 1,247 alerts from a multi‑stage attack scenario, each missing at least 40.4% of attack‑related alerts. Building on these findings, the authors propose AIDA, a multi‑agent framework that enforces explicit decision proposals, independent challenges, and stronger evidence requirements, achieving an F1 score of 0.958 and reducing false‑negative alerts from 40.4% to 3.1% while escalating 18.4% to analysts.
arXiv:2608. 03591v1 Announce Type: cross Abstract: Large Language Model (LLM) agents offer a promising approach to attack chain reconstruction by retrieving and interpreting heterogeneous telemetry to infer ordered attacker actions.
arXiv:2607. 26791v1 Announce Type: cross Abstract: Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities.
Large language model (LLM) agents could help security operations centers (SOCs) investigate advanced persistent threats (APTs) by turning weak leads into evidence for intrusion scoping and response. Y...
arXiv:2606. 18325v1 Announce Type: cross Abstract: Enterprise intrusion response still depends on static playbooks and analyst-driven triage, creating delay between alert generation and containment.
AgentEval is a system that tracks a contract-bound assessment record throughout a stateful agent’s tool execution, capturing state, events, and evidence to determine whether obligations are satisfied, violated, or unresolved. In 32 tool‑world administrations, independent checks confirmed all 30 complete episodes, while controlled interventions on 48 evidence views uncovered 36 binding faults that hash‑only checks missed. A public‑benchmark study on 192 held‑out executions identified ten completed subgoals within broader task failures and highlighted the impact of missing evidence on overall task success.
APTInvestBench is a benchmark that evaluates how well autonomous agents can investigate advanced persistent threats (APTs) when faced with different telemetry settings. It contains 370 cases derived from 56 attack reconstructions, totaling 16.4 million log records, and tests agents on seven SOC-inspired telemetry conditions. The benchmark measures evidence acquisition and formal citation support, revealing that while overall coverage drops only slightly when telemetry is limited, a significant portion of actions lose sufficient citation support, highlighting instability in agent performance.
arXiv:2606. 04990v1 Announce Type: cross Abstract: Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents.
arXiv:2606. 10747v1 Announce Type: new Abstract: As AI systems built from multiple language-model agents become more common, they are increasingly used to make decisions together: discussing, negotiating, and acting on shared tasks.
The paper introduces a cost‑aware Hierarchical Multi‑Agent System (HMAS) for ransomware detection and family attribution that prioritizes static analysis and escalates to dynamic and memory analysis only when necessary, thereby reducing analysis time and resource usage. In experiments on 12,439 samples from 16 ransomware families, the deterministic HMAS achieved high F1 scores (0.93) while resolving nearly 58% of cases with static evidence alone and cutting average internal analysis cost by 44.6% compared to exhaustive methods. The system also records a complete provenance trace for each decision and includes optional local LLM review for limited verdict adjustment.
arXiv:2605. 12729v2 Announce Type: replace-cross Abstract: Large language models are increasingly being used to support network operations (NetOps) and artificial intelligence for IT operations (AIOps), including incident investigation, root-cause analysis, configuration synthesis, and limited self-healing.
arXiv:2606. 22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence?
Diviner is an agentic root‑cause analysis system designed for large‑scale distributed infrastructures. It organizes incident investigation into a four‑stage reliability lifecycle, leveraging a corpus of reviewed incidents and service‑owner expertise to create scoped, versioned skills, workflows, typed tools, and an incident knowledge graph. The system employs adaptive lead and context‑isolated workers to apply relevant knowledge to real‑time evidence, evaluates investigation trajectories against responder‑reviewed outcomes, and updates operational knowledge through feedback loops that require replay, quality checks, and owner approval.