Tool use, function calling, orchestration and the protocols that let models act rather than only answer.
arXiv:2610.33684v1 Announce Type: new Abstract: Self-evolving training improves the performance of agents on difficult tasks through verifiable feedback, but it often applies costly deliberation unif...
AdaPRA is a process reward agent that uses adaptive rubrics to supervise data science agents, correcting errors such as insufficient exploration, unsound validation, coding bugs, and inefficient budget use. It accumulates knowledge across multiple attempts on the same task and adapts its rubrics to each new task, improving performance on DSPredict-Easy and DSPredict-Hard benchmarks. The approach outperforms existing methods like DataPRM and demonstrates that rubrics mined from one model can benefit another, with code released for reproducibility.
BenchBench‑Cloning is a new benchmark for evaluating large language models on end‑to‑end molecular cloning tasks. It contains 65 tasks drawn from real sequence records, laboratory notebooks, and published studies, split into 50 protocol‑development tasks and 15 construct‑design tasks. Performance is measured by simulation execution for protocols and deterministic or rubric‑based checks for design, with GPT‑6 Astra achieving the highest score of 30.6% among eight tested systems.
The paper introduces ConMem, an algorithm that uses counterfactual causal inference to improve long‑term memory management in large language model agents. It builds a benchmark called MemAlign with 1,079 curated memory cases that capture various misalignment issues such as over‑generalization, misattribution, and privacy concerns. ConMem generates counterfactual interaction scenarios to audit retention decisions, leading to higher accuracy (75.16%) and better balance than existing methods while preserving personalization.
SkillShift-Bench is a new benchmark that evaluates web agents’ skill learning under controlled open-environment shifts, including Contextual, Perceptual, and Execution shifts across three self-hosted web systems. It measures task success, interaction cost, skill use, and skill failure, revealing that skill robustness degrades systematically—especially under Execution Shift. The benchmark also frames shifted skill execution as a nonstationary local executability problem, offering a dynamic-regret analysis and a test-time adaptation method to improve skill execution without relearning.
The paper introduces System 1 models—fast, single‑pass decision engines that map inputs directly to structured outputs, addressing the latency, cost, and calibration issues of large language models. It presents a five‑family taxonomy (encoder‑based classifiers, reward models, non‑autoregressive engines, contrastive models, and extraction models) and discusses training, deployment trade‑offs, and calibration theory, including proper scoring rules and RLCD. The guide also covers integration into agentic workflows, System 1+System 2 cascades, 18 application domains, an open‑source ecosystem, and practical resources such as code, a model zoo, and assessment questions.
ReGE is a framework that transforms recorded interactive task experience into guidance for new attempts. Its implementation, ReGE‑Syn, synthesizes execution records and artifacts into concise, self‑contained instructions using a single coach call, without updating model parameters. Experiments on four benchmarks show that ReGE improves accuracy by roughly 9–10 percentage points for GPT and Claude, and significantly reduces guidance‑construction tokens while maintaining success rates.
MatOS is an agent operating system designed to streamline materials research by organizing data, knowledge, tools, and workflows into reusable assets. It features a layered scientific asset graph, a stateful adaptive workflow that tracks and resumes tasks, and an evidence-driven asset evolution mechanism that updates assets based on execution records and user feedback. In benchmark tests, MatOS outperforms existing agents and has successfully narrowed millions of candidate structures to a few promising solid-electrolyte and halide variants, demonstrating its effectiveness in long, multistage materials studies.
The paper introduces CTXAgent, a self‑controlled context management system for large language model agents that compacts conversation history when a token threshold is reached. It evaluates seven frontier models, comparing their ability to manage context with and without dedicated tools, and benchmarks performance across coding, scientific, and software engineering tasks. Results show that well‑managed models keep live context below 100k tokens, maintain high benchmark scores, and achieve significant cost reductions, with Claude Opus 5.5 outperforming its explicit configuration by 34% lower cost.
The paper investigates how to structure verification in agent harnesses that let large language models use tools and iteratively refine their outputs. It evaluates 17 verification designs across four lifecycle layers—initial instructions, mid‑turn reminders, completion gates, and independent review—on coding benchmarks (Terminal‑Bench and SWE‑bench) and a new VisualBench for visual artifacts. The results show that independent review by a critic consistently improves accuracy with modest cost increases, outperforming other designs and offering a practical strategy for test‑time scaling.
The paper introduces HaMo, a Harness-Model collaborative framework designed for long-horizon continual learning. HaMo combines fast non‑parametric experience accumulation—distilling trajectories into strategy‑level experience cards for immediate retrieval—with slow parametric adaptation, training the model to interpret, ground, and execute these strategies across changing contexts. A retrieval‑coverage‑based selection mechanism identifies compact, complementary trajectories for parameter updates, and experiments demonstrate that HaMo outperforms both harness‑only and model‑only approaches in within‑domain and cross‑domain settings, improving retention and adaptation.
SearchHorizonBench is a new benchmark that evaluates how AI agents decide whether the cost of an additional search is justified by its expected benefit. It provides 45,632 executions on 3,168 questions, using paired continuations and repeated runs to separate true search gain from generation noise. The dataset shows that historical calibration can reduce Brier error by 16.2% and that free‑selection observations offer a 2.49‑point utility advantage over a constant action, highlighting the importance of accurate search‑value estimation.
The paper introduces query-driven learning for lifted reasoning, aiming to combine learning and reasoning into a single process rather than treating them as separate stages. It extends previous work to the first-order setting and identifies a necessary and sufficient condition called bounded concealment for validating or refuting universal queries with partial data and a knowledge base. A quasi-polynomial-time algorithm is proposed, with soundness and completeness guarantees relative to space-bounded lifted proofs, and quasipolynomial sample and runtime bounds are shown to make lifted query-driven reasoning efficient even when unrestricted knowledge-base learning is not feasible.
The paper investigates how to design centralized critics for N-agent ad hoc teamwork, where a learning agent must coordinate with an unknown set of uncontrolled teammates. It compares two advantage estimators: controlled history estimators that use only the learner’s own histories, and joint history estimators that also incorporate the uncontrolled teammates’ histories. The authors prove both estimators are unbiased, show conditions where joint history estimators have lower variance, and demonstrate empirically that joint history critics achieve better sample efficiency and final returns across various NAHT tasks.
arXiv:2610. 34855v1 Announce Type: new Abstract: Visual counting benchmarks report whether vision-language models produce the correct number, but not whether errors arise from perception or reasoning.
PatentAgentBench is a new benchmark that tests large language model agents on 510 real-world patent tasks across eight suites, including novelty search, freedom‑to‑operate, design FTO, invalidity search, claim charting, patent drafting, office‑action response, and patent translation. The benchmark uses curated patent records and expert‑reviewed annotations to evaluate task‑specific metrics, requiring agents to search patent collections without pre‑supplied candidate pools and to produce high‑quality drafting and response work. Seven LLMs were evaluated on four agent harnesses, revealing that current systems still struggle with novelty identification, risk assessment, and understanding practitioner intent throughout patent workflows.
The paper introduces Multiscale Policy Optimization (MPO) for long‑horizon large language model agents, addressing the mismatch between turn‑level critic values and token‑level action generation. MPO combines Multiscale Advantage Estimation (MAE) with turn‑level importance sampling to better align credit assignment and policy updates at the turn scale. Experiments on search, tool‑use, and automation tasks with 4B and 30B‑A3B models show consistent improvements over strong reinforcement learning baselines.
Same Story, Different Lens is an experimental prototype that compares how 17 news outlets cover the same events by collecting world and politics coverage, grouping reports about identical incidents, and generating structured analyses of sentiment, political lean, framing, loaded language, omissions, and estimated proportions of factual claims versus opinion. The system employs three model roles—article analysis, source support assessment, and cross‑outlet briefing generation—to produce JSONL archives of its findings. Evaluation includes comparing outlet‑level lean estimates with predefined reference ratings and summarizing the Critic’s grounding scores, which indicate the model’s assessment of support within the supplied text.
Co-Design Bench is a new, reference‑free benchmark comprising 273 multimodal coding tasks derived from 26,314 real user sessions. It features two tracks—Multimodal Editing and Repository Editing—that test agents on localized edits with optional image, audio, video, or document inputs and coordinated repository changes across three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The benchmark uses a judge‑then‑refute protocol with hidden expert‑authored acceptance criteria, and the best of ten model endpoints achieves an average of 69.9% criterion completion but fully satisfies only 26.4% of tasks, especially struggling with repository edits involving multiple change types.
AgenticCity-Bench is a longitudinal multimodal benchmark that unifies state modeling and agentic reasoning for urban dynamics. It covers 16 cities across 12 countries from 2000 to 2025, providing 18 modalities that capture urban function, activity, and environmental state. The benchmark aligns heterogeneous data around a common spatial reference and calendar index, enabling cross‑modal translation and diverse agentic city tasks such as forecasting, planning, and resource analysis.