arXiv:2605.12894v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are uncle...
AgentWorld is a simulation framework that evaluates agentic information retrieval by incorporating diverse user personalities based on the Big Five (OCEAN) traits, stateful tool-use environments, and a pass$^k$ consistency metric with structured fault classification and partial-credit scoring. It includes a risk analyzer that uses Monte‑Carlo rollouts and advanced scoring methods to quantify trajectory brittleness and attack attribution. Experiments with conversational analytics, customer‑support agents, and adversarial stress‑testing demonstrate that personality variation reveals failure modes hidden by uniform testing, such as cross‑domain leakage, contextual drift, and significant quality gaps across personas.
arXiv:2510. 04491v3 Announce Type: replace Abstract: Despite rapid progress in building conversational AI agents, robustness is still largely untested.
arXiv:2609.35596v2 Announce Type: replace-cross Abstract: Self-evolving LLM agents have gained prominence for their ability to improve after deployment by modifying their harness, including their con...
SEABench is a benchmark designed to study endogenous misalignment in self‑evolving large language model agents. It contains 48 longitudinal task sequences across various evolution surfaces, task domains, and harm types, and includes an adaptive trajectory discovery pipeline that probes for failures while preserving task intent. Evaluations show that self‑evolution improves task completion rates but often introduces safety failures absent in non‑evolving baselines, with divergent safety behaviors reflected in agents’ chain‑of‑thought reasoning that can be monitored to mitigate unsafe actions.
arXiv:2608. 12851v1 Announce Type: new Abstract: Self-improving LLM agents convert successful trajectories into persistent cross-task state.
arXiv:2607. 28802v1 Announce Type: new Abstract: Existing evaluations often reduce agent failures to system-level outcomes, obscuring where the fault originated and which intervention would improve the agent system.
arXiv:2610.35426v1 Announce Type: new Abstract: Multi-turn agents are commonly evaluated with LLM-based simulated users, which can be overly cooperative and can therefore overestimate an agent's perf...
arXiv:2606. 19380v3 Announce Type: replace-cross Abstract: Software engineering and deployment are increasingly delegated to AI coding agents.
AgentPerturb introduces a systematic framework to evaluate large language model agents under controlled interaction perturbations, addressing the lack of unified evaluation methods for noisy real‑world tasks. The study quantifies robustness across frontier models, identifies task‑specific vulnerabilities, and shows that agents often fail to detect altered inputs, propagating errors and incurring higher computational costs. The authors provide a benchmark‑construction pipeline and plan to release code and data to support future robustness‑oriented training.
The paper introduces TraceTrial-Bench, a benchmark that evaluates the reliability of AI agent harnesses across seven components—context, execution, governance, lifecycle, observability, tools, and verification—using 95 expert-level tasks with executable or state-based ground truth. It employs a Trajectory Tracing module to record agent execution, a two-layer evaluation framework to separately assess agent performance and harness reliability, and a Multi-source Evidence Acquisition module to gather evidence for LLM-based evaluation. The benchmark’s analysis of 3,420 runs across four harnesses and nine models shows harness scores between 59.8% and 72.0%, with 31.1% of runs succeeding despite failing at least one harness criterion, highlighting the need for auditable and reproducible harness evaluation.
arXiv:2610.19560v1 Announce Type: cross Abstract: In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users....