arXiv AI
NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 2
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
UniACE is a unified framework that standardizes the evaluation of large language model (LLM) agents by representing each benchmark as an instruction–tool–environment triplet and running models through a shared, task‑agnostic harness in isolated runtimes. It preserves native success criteria, offers an offline mode for dynamic‑resource tasks, and standardizes efficiency metrics, execution records, and failure attribution. Applying UniACE to 7 benchmarks across 24 domains and 15 models revealed significant score shifts, ranking reversals, and sensitivity to evidence representation, highlighting the impact of evaluation configuration on reported agent performance.
arXiv AI
Oct 15
ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?
ProtoArena is a new black-box benchmark that tests AI agents on 80 cross‑component vulnerability‑exploitation environments across 11 protocol and application domains and 7 attack categories. It evaluates agents at three information levels—guided reproduction, exploitation with context, and effect‑directed black‑box exploration—using a hybrid verifier that combines rule‑based checks with LLM review of traffic and logs. In experiments, Claude Opus 4.8 achieved the highest success rates, reaching 48.8% under guided conditions and 12.5% in effect‑directed black‑box scenarios, while also demonstrating significant DNSBomb optimization performance.
Hugging Face Trending Papers
Oct 13
ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?
Large language model (LLM) agents are increasingly used for vulnerability analysis, yet existing benchmarks provide limited coverage of exploitation that depends on interactions among network protocol...
arXiv AI
Jul 16
AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
arXiv:2607. 13705v1 Announce Type: new Abstract: As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical.
arXiv AI
Oct 16
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
Harness-Bench is a diagnostic benchmark that evaluates how different harness configurations affect LLM agent performance in realistic workflows. It tests 106 sandboxed offline tasks across multiple model backends, recording artifacts, execution traces, and usage statistics to analyze completion, process quality, efficiency, and failure behavior. The study finds significant variation in outcomes across model-harness pairings, indicating that agent capability should be reported at the configuration level rather than solely by the base model.
arXiv AI
Jun 6
Evaluating Agentic Configuration Repair for Computer Networks
arXiv:2606. 06212v1 Announce Type: new Abstract: Misconfigurations in computer networks remain a major source of critical Internet outages.