The paper proposes a framework for agent calibration that ensures consistent capabilities when agents are deployed, migrated, or scaled across different models, harnesses, infrastructures, and user contexts. It defines standards for basic capabilities, technical environments, and user contexts, diagnoses gaps, applies revisions, and rechecks within fixed budgets, using diagnostic source behavior and mandatory tests to qualify agents. The framework includes semantic checkpoints, repair localization, and reusable configuration artifacts, and outlines an automatic calibration tool that relies on limited authorized user trajectories and tests, though empirical validation remains pending.
By Yaxiao Liu (PwC China AI Center), Pengbo Liu (PwC China AI Center), Yiwen Liu (PwC China AI Center), Yihua Guan (PwC China AI Center), Jiaxing Song (Tsinghua University)
The paper introduces ACES (Agentic Continuous Evaluation of Skills), a framework that evaluates reusable skills and capability packages by running paired live trials with and without a target skill, normalizing results into the Agent Trajectory Interchange Format (ATIF), and grading six runtime metrics to compute Skill Lift. ACES demonstrates that scan-only gates miss important aspects of skill performance, while the evaluation protocol reveals significant improvements in skill execution, behavior check, and skill efficiency across 145 real skills and 947 scored cases. The open‑source NVIDIA SkillEvaluator implementation enables reproducible, repository‑native assessment of agentic skills in production environments.
By Christopher Kevin, Narendran Raghavan, Jean-Francois Puget, Roshni Malani, Meghana Puvvadi, Moshe Abramovitch, Mohit Gupta, Rama Akkiraju, Subodh Prabhu, Yogesh Dangi, Wei Luo, Seong Hee Lee
arXiv:2610.08875v1 Announce Type: cross
Abstract: Agent Skills package procedural guidance and resources for reuse, but a relevant Skill does not necessarily improve task performance. Existing studie...
By Yu Cheng, Dehai Zhao, Zhongxin Liu, Qing Huang, Zhenchang Xing, Xiaoxue Ren
Praxa is an evidence‑bound harness for governed AI agent execution that explicitly represents states such as proposal, authority, dispatch, verified external effect, and promotion through deterministic admission, brokered execution, external read‑back, reconciliation, and reviewed promotion. The authors report four evidence lanes: a repository‑local audit passing all unit and Workerd tests; a pilot on 12 curated tasks where both baseline and reliability‑layer arms passed 17 of 36 trials; a coordination‑proxy comparison where both baseline and a source‑authored candidate completed all 180 trials with equal accuracy but the candidate used fewer tokens and steps; and deployed source/configuration evidence showing bounded reflection, recall accounting, memory compilation, and tool‑health paths. None of the evidence demonstrates superiority in security, safety, or user benefit.
whyItMatters:"Praxa provides a testable architecture that makes authority‑to‑effect transitions explicit, offering a framework for verifying AI agent behavior, though current evidence does not prove improved security or performance."
By Stefan G. Creadore
arXiv:2607. 25398v1 Announce Type: new Abstract: Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows.
By Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta, Edwin Chen
The paper investigates how language‑model judges can make version‑dependent errors when evaluating upgraded agents. Using 35 public coding‑agent submissions, two customer‑service agents, and over a thousand expert‑labeled trajectories, the authors show that fixed judges often reject task‑conditioned error invariance and can incorrectly approve failed patches, especially as agent capability increases. Paired audits of current outputs reduce interval width only marginally, and the study concludes that independent human patch review is still necessary.
By Jiapeng Li
arXiv:2605. 22949v3 Announce Type: replace Abstract: Foundation-model pools are increasingly used as black-box responders in coordinated systems where a coordinator must decide which response to trust.
By Joss Armstrong
arXiv:2610.17757v1 Announce Type: new
Abstract: Large language model agents can follow complex human instructions to solve real-world tasks. In practice, however, initial instructions are often incom...
By Kaiming Liu, Yuxuan Liu, Jinrui Ju, Minyuan Ruan, Xuanyu Lei, Peng Li, Yang Liu
The paper introduces NAQD‑Env, a synthetic benchmark designed to test language agents’ ability to selectively withdraw and resume actions when new evidence, permissions, or stop instructions arise. It evaluates models against a deterministic reference policy across eleven dependency families, measuring policy agreement, task value, withdrawal, resumption, and event reporting. Experiments on 350 scenarios show low withdrawal recall, no valid resumption, and limited policy alignment, highlighting the need to treat selective withdrawal as a distinct reliability component.
By Mohamed Abouzahra
arXiv:2609.21841v1 Announce Type: new
Abstract: Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically val...
By Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona Zahid
arXiv:2610.14918v1 Announce Type: new
Abstract: Evaluating an agent's ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability....
By Zhengxi Li, Junyi Zhou, Qiyuan Zhang, Chuhan Wu, Fuyuan Lyu, Le Tian, Xiao Zhou, Chen Ma, Yufei Wang
The paper investigates how fine‑tuned language agents can selectively withdraw actions when evidence changes or authority is revoked. Using a synthetic NAQD‑Env environment, the authors compare an original curriculum with an enriched one across multiple system‑prompt framings and evaluate four LoRA checkpoints of Qwen2.5‑3B‑Instruct against instruction‑tuned baselines. Results show that fine‑tuned checkpoints achieve higher decision agreement and withdrawal recall, reduce incorrect withdrawals, and narrow prohibited combinations, though they also omit valid event reports and exhibit mixed preservation of authorized actions.
By Mohamed Abouzahra