R2A‑Sales is a new benchmark that pairs static policy probes with multi‑turn sales scenarios to evaluate whether large‑language‑model agents can translate stated rules into compliant, tool‑grounded actions under commercial pressure. The benchmark includes a ledger‑verified interactive arm, 24 policy atoms, 42 semantic cores, and 2,016 matched pairs across three Chinese‑language sales domains. Results show high static compliance (≈90–99%) but low action compliance (≈3–17%), revealing a substantial rule‑to‑action gap of 80–92 percentage points.
arXiv:2608. 12599v1 Announce Type: new Abstract: Multi-turn dialogues let users revoke constraints as easily as impose them, but revocation does not reliably take effect: models keep enacting withdrawn requirements (occasionally beneath comments asserting their removal), a failure we call \emph{behavioral relapse}, or revocation inertia.
arXiv:2608. 12476v1 Announce Type: new Abstract: Long-term agent memory is usually treated as select--store--retrieve, but retrieval does not decide whether contradictory, superseded, retracted, deleted, or stale records may support an outgoing claim.
The paper introduces a framework for evaluating how large language model agents revise their success criteria after failures, defining five non‑compensatory conditions that must be met for a criterion revision to be considered valid. Using the CMB‑0.1 protocol, the authors test twelve cross‑domain scenarios across four system configurations, finding that no model trial satisfies all five conditions and highlighting specific failure modes such as zero‑state reconstruction and inadequate intervention sensitivity. They propose a more stringent trace‑anchored CMB‑0.4 protocol to better isolate and measure criterion revision in future studies.
arXiv:2607. 23386v1 Announce Type: new Abstract: We document a failure class in frontier large language models -- exception chain collapse -- observed in eligibility evaluation under nested conditional rules of the form "A is required UNLESS B applies, UNLESS C overrides B".
arXiv:2606. 29654v1 Announce Type: new Abstract: Multi-agent deliberation among LLMs can improve reasoning, but deployment requires deciding when the current answer is reliable enough to act on and when it should be escalated to human review.
CRM‑Agent‑Bench is a benchmark that evaluates language agents in customer‑relationship‑management settings by testing 555 stateful, multi‑turn tasks across nine business domains. Each task includes an executable contract with checks on actions, arguments, order, responses, target state, and protected state, and success requires meeting all these checks. The benchmark reports coverage, latency, and cost, showing that evaluated systems achieve 64.32%–83.60% strict success while varying in cost and domain reliability.
The paper investigates how language‑model judges can make version‑dependent errors when evaluating upgraded agents. Using 35 public coding‑agent submissions, two customer‑service agents, and over a thousand expert‑labeled trajectories, the authors show that fixed judges often reject task‑conditioned error invariance and can incorrectly approve failed patches, especially as agent capability increases. Paired audits of current outputs reduce interval width only marginally, and the study concludes that independent human patch review is still necessary.
arXiv:2610.34167v1 Announce Type: new Abstract: Tool-using language model agents are usually evaluated by whether they finish the user's task. That metric misses a common failure: an agent can comple...
arXiv:2607. 25398v1 Announce Type: new Abstract: Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows.
arXiv:2609.26048v1 Announce Type: cross Abstract: Language-model agents often reach a working solution and then fail to consistently deliver it. We study runtime policies: targeted natural-language i...
arXiv:2607. 28685v1 Announce Type: new Abstract: Agent-safety benchmarks measure different behaviors, and their scores get quoted interchangeably as an agent's safety.
arXiv:2607. 03386v1 Announce Type: new Abstract: Agentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert action labels are unavailable.