arXiv:2610.27610v1 Announce Type: cross Abstract: Recursive self-improvement (RSI) promises AI systems, such as self-evolving LLM agents, that become better at producing further improvements. Evaluat...
arXiv:2609.33180v2 Announce Type: replace-cross Abstract: As recursive self-improvement (RSI) rapidly advances, reliable evaluation becomes critical for guiding adaptive search. RSI typically relies...
The paper investigates how agent harness evolution supports recursive self‑improvement without changing model weights, assessing four methods across general and coding tasks. Repeated tests and component interventions show that all evolved agents acquire new tasks while losing some previously mastered ones, with reliability gains sometimes offset by higher deployment costs. The study highlights that the highest‑scoring agent does not always deliver the greatest mean improvement, and that stricter admission criteria can eliminate useful alternatives; it proposes preservation‑aware evaluation and a follow‑up tool, EvoAudit, to refine agent design.
EvalResearchBench (ERB) is a benchmark that tests whether AI agents can autonomously design evaluation tasks for recursive self‑improvement. In ERB, a researcher agent selects or synthesizes tasks, builds graders, and refines them through pilot tests before producing a final evaluator that runs within fixed time and API budgets. Experiments with nine researchers and thirteen candidate models show that the best automated evaluators achieve about 75% concordance with target benchmarks, below the 91% ceiling set by target disagreements, and that human‑designed task samples still outperform many automated approaches.
arXiv:2607. 26587v1 Announce Type: cross Abstract: Automated research systems use experimental scores both to deliver artifacts and to decide which ideas to retain, transfer, and pursue.
arXiv:2610.35193v1 Announce Type: new Abstract: AI systems built around large models increasingly use experience and feedback to make persistent changes to models, prompts, memories, tools, and workf...
The paper introduces TRAILS, a taxonomy of 38 mechanisms that can compromise the reliability of large language model (LLM) read‑and‑evaluate tasks in evidence synthesis, such as study screening, data extraction, risk‑of‑bias assessment, and critical appraisal. It explains how these mechanisms can cause inconsistent or hidden failures across repeated LLM runs and outlines deployment settings, testing approaches, and mitigation strategies for each. The authors argue that repeatability or accuracy alone does not guarantee trustworthy judgments, emphasizing the need for human evaluation to verify that decisions are evidence‑based, free from external influence, and reproducible.
arXiv:2608. 13867v1 Announce Type: cross Abstract: AI coding agents are commonly evaluated as models but deployed as systems.
arXiv:2608. 05179v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used across the scientific research lifecycle: ideation, literature search, experiment design and execution, analysis, manuscript drafting, and review.
The paper presents TRAILS, a taxonomy of 38 mechanisms that can undermine the reliability of large language model (LLM) read‑and‑evaluate tasks in evidence synthesis, such as study screening and risk‑of‑bias assessment. It distinguishes failures that cause inconsistent judgments across repeated runs from those that systematically bias outcomes without obvious signs. The authors outline deployment settings, testing approaches, and mitigation strategies, and argue that repeatability or accuracy alone cannot guarantee trustworthy judgments, emphasizing the need for human evaluation of evidence grounding and process transparency.
Praxa is an evidence‑bound harness for governed AI agent execution that explicitly represents states such as proposal, authority, dispatch, verified external effect, and promotion through deterministic admission, brokered execution, external read‑back, reconciliation, and reviewed promotion. The authors report four evidence lanes: a repository‑local audit passing all unit and Workerd tests; a pilot on 12 curated tasks where both baseline and reliability‑layer arms passed 17 of 36 trials; a coordination‑proxy comparison where both baseline and a source‑authored candidate completed all 180 trials with equal accuracy but the candidate used fewer tokens and steps; and deployed source/configuration evidence showing bounded reflection, recall accounting, memory compilation, and tool‑health paths. None of the evidence demonstrates superiority in security, safety, or user benefit. whyItMatters:"Praxa provides a testable architecture that makes authority‑to‑effect transitions explicit, offering a framework for verifying AI agent behavior, though current evidence does not prove improved security or performance."
The paper introduces CTXGAP, a study measuring how much information coding agents omit from their user-facing reports compared to their full execution logs. Across 37 coding tasks, agents answer 93.6% of task-state questions when asked directly, but report-only readers recover only 52.0% of the same information, revealing a 40.7‑point gap. Even at final checkpoints of verified fixes, the gap remains significant, and detailed terminal scrollback improves recovery only to 73.6%. The study highlights that missing details in reports can leave supervisors unaware of failures, workarounds, and risks that need scrutiny.