arXiv AI
Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 26
AgentWorld: Personality-Aware Reliability Evaluation for Agentic Information Retrieval
AgentWorld is a simulation framework that evaluates agentic information retrieval by incorporating diverse user personalities based on the Big Five (OCEAN) traits, stateful tool-use environments, and a pass$^k$ consistency metric with structured fault classification and partial-credit scoring. It includes a risk analyzer that uses Monte‑Carlo rollouts and advanced scoring methods to quantify trajectory brittleness and attack attribution. Experiments with conversational analytics, customer‑support agents, and adversarial stress‑testing demonstrate that personality variation reveals failure modes hidden by uniform testing, such as cross‑domain leakage, contextual drift, and significant quality gaps across personas.
arXiv Computation and Language
6d ago
On the Behavioral Traits of LLM Agents
The paper introduces A-B-D, a method that infers personality traits of large language model agents from their behavioral data, combining functional actions and linguistic outputs across 345,667 real-world trajectories of 80 models. After filtering for stability and discriminability, 79 features are reduced to six reliable factors—two functional and four linguistic—revealing differences such as Claude‑Sonnet‑4.6’s planfulness versus GPT‑5.5’s low energy. The study also demonstrates a weak correlation between these inferred traits and agents’ self-reported Big Five scores, highlighting a knowledge‑action gap.
arXiv AI
Jun 11
Preregistration for Experiments with AI Agents
arXiv:2606. 11217v1 Announce Type: cross Abstract: The proliferation of large language models (LLMs) and autonomous AI agents has given rise to a rapidly growing methodological paradigm: "in silico" behavioral experiments.
arXiv AI
Sep 1
Candidate supply and answer selection shape the value of LLM judging in multi-agent systems
arXiv:2608.25937v2 Announce Type: replace Abstract: Multi-agent systems (MAS) sometimes already have the potential to answer correctly, but still report a wrong answer. Explaining this outcome is dif...
arXiv Computation and Language
Oct 22
SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges
SANE is an agent‑based framework that automatically generates scorer‑specific perturbation recipes and expected score relations for large‑language‑model judges, using behavioral requirements encoded as reusable agent instructions. It validates each recipe against the scorer’s intent and the instantiated test case, producing a growing inventory of aligned recipes as more scorers are added. In experiments across five domains, SANE yields 3.7× more validated test families than the Judge Reliability Harness, detects deficiencies in all domains—including professionalism and toxicity—and achieves strong human alignment in judge ranking on faithfulness (Spearman ρ = 0.852).
arXiv AI
Aug 26
Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight
The paper evaluates the use of large language models (LLMs) as judges for assessing conversational voice agents, comparing human judgments with GPT‑4.1 and GPT‑5 across telecom and retail interactions. It examines agreement, metric‑level correlations, and consistency across three evaluation configurations (p0, p1, p2) to determine how reliably LLMs can judge conversational quality and safety. The study finds that LLM‑based evaluation can be effective but its reliability varies by metric and configuration, suggesting a hybrid approach where LLMs handle scalable assessment while humans focus on metrics requiring contextual interpretation.