EIO-Agents introduces an open specification for interoperable AI agent evaluation, consisting of two layers: the Evaluation Intelligence Ontology (EIO) and the Portable Evaluation Record (PER). The EIO layer supplies a semantic framework that defines typed evidence, behavioral predicates, evidence contracts, claims, witness rules, proof status, recurrence, and computable derivations for metrics, findings, controls, and decisions such as PASS, REVIEW, or BLOCK. The PER layer offers a canonical, content‑addressed record of an evaluation that preserves the evidence‑to‑decision chain, enabling re‑derivation, explanation, and verification of the evaluation outcome.
arXiv:2606. 04990v1 Announce Type: cross Abstract: Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents.
arXiv:2606. 04990v2 Announce Type: replace-cross Abstract: Large language model (LLM)-based agents are evolving from passive text generators into autonomous systems capable of planning, tool use, retrieval, memory access, environmental interaction, and multi-agent collaboration.
arXiv:2606. 09809v1 Announce Type: new Abstract: AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs.
arXiv:2610.18603v1 Announce Type: new Abstract: Benchmarks report what an Artificial Intelligence (AI) agent can do on a task suite, but not whether one released version has been engineered with the...
AgentEval is a system that tracks a contract-bound assessment record throughout a stateful agent’s tool execution, capturing state, events, and evidence to determine whether obligations are satisfied, violated, or unresolved. In 32 tool‑world administrations, independent checks confirmed all 30 complete episodes, while controlled interventions on 48 evidence views uncovered 36 binding faults that hash‑only checks missed. A public‑benchmark study on 192 held‑out executions identified ten completed subgoals within broader task failures and highlighted the impact of missing evidence on overall task success.
arXiv:2609.21841v1 Announce Type: new Abstract: Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically val...
arXiv:2608. 07688v1 Announce Type: new Abstract: IT audits require auditors to judge whether heterogeneous organizational evidence satisfies semantic security and compliance controls.
arXiv:2609.24755v1 Announce Type: new Abstract: Autonomous AI agents are increasingly deployed in areas where wrong decisions are hard to reverse. This paper examines schema mismatch: the condition i...
CARGO is a framework for evaluating agentic AI systems in production that addresses the problem of reference-instance divergence (RID), where reference-based judges penalize correct answers that involve different entity identifiers. It treats retrieved references as procedural exemplars, grounds judgments in the live instance’s context, assigns a three-way status to claims, and gates evaluation by retrieval confidence. Using the CARGO-Bench diagnostic suite, CARGO eliminates false penalties and improves discrimination while revealing a limitation in detecting procedural corruptions.
arXiv:2606. 22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence?
arXiv:2606. 16603v1 Announce Type: cross Abstract: LLM-based agents have demonstrated strong capabilities in data-intensive analytical tasks, yet their outputs are rarely verifiable: a reliance on linear text trajectories makes their reasoning difficult to audit.
AuditKIT is a source‑available evaluation library that unifies benchmark accuracy, judge‑based assessment, runtime performance, tool‑use behavior, and retrieval quality into a single result representation. It offers direction‑aware metric contracts, configuration fingerprints, and extensible metrics and backends, while providing turn‑aware tool‑call matching and distinguishing verified task outcomes from trajectory‑level diagnostics. Seven case studies demonstrate how answer extraction, runtime behavior, and metric selection can alter evaluation outcomes across model comparison, serving runtimes, retrieval, tool calling, and interoperability.