arXiv:2609.25237v1 Announce Type: new Abstract: Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned,...
arXiv:2610.03448v1 Announce Type: cross Abstract: LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchma...
Praxa is an evidence‑bound harness for governed AI agent execution that explicitly represents states such as proposal, authority, dispatch, verified external effect, and promotion through deterministic admission, brokered execution, external read‑back, reconciliation, and reviewed promotion. The authors report four evidence lanes: a repository‑local audit passing all unit and Workerd tests; a pilot on 12 curated tasks where both baseline and reliability‑layer arms passed 17 of 36 trials; a coordination‑proxy comparison where both baseline and a source‑authored candidate completed all 180 trials with equal accuracy but the candidate used fewer tokens and steps; and deployed source/configuration evidence showing bounded reflection, recall accounting, memory compilation, and tool‑health paths. None of the evidence demonstrates superiority in security, safety, or user benefit. whyItMatters:"Praxa provides a testable architecture that makes authority‑to‑effect transitions explicit, offering a framework for verifying AI agent behavior, though current evidence does not prove improved security or performance."
The paper investigates why training terminal agents often stalls when using a meta‑agent like Claude Opus to generate tasks and verifiers. It identifies three failure modes—benchmark invalidity, harness brittleness, and reward misalignment—and shows that prompt redesign and context extension can improve solvability by 5.6×, yet a 9B model still tops out at 81.3% mean pass@2. Adding hard tasks drops performance to 20.6%, highlighting that solvability depends on the model used.
TRACE (TRAjectory-Contrastive Evolution) is a self‑evolving skill bank that improves the consistency and limit‑awareness of large‑language‑model agents without changing the model weights. By iteratively refining modular skills based on successful and failed trajectories, TRACE raises consistent performance (Pass^3) on the CAR‑bench in‑car assistant tasks from 59.9 % to 94.5 % on GPT‑5.5 and achieves first place on the hidden set with GPT‑5.6‑Sol. The approach demonstrates that a skill‑based, self‑evolution loop can convert a model’s potential into stable, reliable behavior.
The paper introduces Traverse, a benchmark of 2,518 agent trajectories and 6,967 annotated mistakes across software engineering, computer use, and science tasks, revealing that failures often go unrecovered and can cause irreversible harm before a run is deemed successful. It shows that human judges struggle to detect the first mistake in most runs, while a 4‑billion‑parameter verifier called Scout can locate failures more effectively and improve task success when used to select among candidate runs. The study demonstrates that making failure detection inexpensive and reliable can enable long‑horizon agents to learn from their own mistakes and increase trustworthiness in autonomous AI.
arXiv:2610.02267v1 Announce Type: new Abstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input...
arXiv:2602. 11619v2 Announce Type: replace Abstract: Running the same LLM agent on identical inputs yields 2.
arXiv:2607. 25152v1 Announce Type: new Abstract: Long-running autonomous agents plan, act, and judge their own completion without human intervention.
AutoTuneBench introduces a trustworthy measurement protocol for evaluating how large language model agents auto‑tune GPU kernels and serving engines. The benchmark addresses four failure modes—strawman baselines, machine‑dependent timing, saturated tasks, and infrastructure defects—by enforcing code‑frozen protocols, database validation, anti‑cheat checks, pre‑registered comparisons, and external result anchoring. Using this protocol, the authors demonstrate that previously reported speedups are inflated, revealing more modest improvements across different engines and machines.
AgentAudit is an open, extensible framework that evaluates the full lifecycle of AI agents, assessing planning, tool selection, execution, memory, and reasoning across ten dimensions such as instruction integrity, security, and alignment. Unlike existing benchmarks that focus on single aspects, AgentAudit analyzes the entire execution trace to attribute failures to specific stages. The framework was tested on five large language models, revealing significant differences in trustworthiness even among models with similar task‑completion performance.
The paper introduces a disclosure‑gated user simulator for companion‑agent evaluation, where information release is conditioned on the agent’s behavior through a five‑level gate system. The authors train the simulator on synthetic and real data, audit its performance, and demonstrate that it preserves ranking order and score stability across 12 tested systems. Their released simulator achieves a 0.993 correlation with the original benchmark’s simulator and outperforms prompting a frontier model, which only shifts scores upward without affecting rankings.