arXiv AI By Leon Goldberg, Gal Engelberg, Eden Yavin, Elad Elouz, Ariel Zadok, Konstantin Koutsyi

Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigations

Read the original on arXiv AI →

The article evaluates the Sola Security Brain, a purpose-built security intelligence layer, against a general-purpose coding agent (Claude Code) on 28 cloud‑security investigation tasks. The Sola Security Brain achieved 0.693 coverage versus 0.387 for the coding agent, a 79.2% relative gain, and outperformed the agent on 25 of 28 tasks while incurring far lower reasoning and cost per unit of coverage. The study also identifies a ‘sample‑and‑generalise’ pattern where the live agent reports universal negatives based on limited sampling, illustrating a potential efficiency trade‑off in cloud investigations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Aug 28

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

BekchiAI introduces a benchmark and platform for evaluating large language model agents. The benchmark comprises 13 tool‑using ReAct agents across seven task categories, totaling 2,057 deterministic test tasks with verifier‑checkable gold answers. The platform offers web‑based observability, token and latency telemetry, and remote run termination for live agents.

By Mesut Toruk
arXiv AI
Jul 31

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

arXiv:2607. 26791v1 Announce Type: cross Abstract: Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities.

By Lehan Wang, Boli Chen, Ruixue Ding, Pengjun Xie, Jinwei Huang, Zhendong Liu, Shuo Wang, Tao Lei, Xin Ouyang, Xiaomeng Li