AgentHop is a diagnostic benchmark comprising 1,011 multiple‑choice questions that test agentic multi‑hop question answering over scientific citation graphs. It provides a controlled seven‑tool sandbox with fixed token, turn, and tool‑call limits, allowing researchers to dissect model performance across four operational axes: retrieval, synthesis, tool‑call, and resource management. Analysis of 19 models shows distinct family‑specific behaviors, such as GPT models committing early, Anthropic and GLM checkpoints verifying before committing, DeepSeek and Kimi over‑searching, and Gemini‑3 Pro maintaining balance, while within‑family differences are revealed in retrieval versus synthesis strategies.
arXiv:2610.32441v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activitie...
arXiv:2607. 05775v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and operate over extended horizons.
Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and operate over extended horizons. Reported benchmark gains often obscure recurring failure modes documented across otherwise unrelated evaluation efforts.
The paper introduces BRIA (BRanch-Inspect-Adopt), an agent that isolates tool‑candidate investigations into separate branches, executing each candidate independently and then inspecting the execution evidence to adopt only useful results. BRIA dynamically retrieves additional tools as new needs arise during task execution and is evaluated on PlanBench‑XL and MCP‑Atlas with a library of 1,897 tools, outperforming or matching the strongest retrieval baselines. On MCP‑Atlas, BRIA achieves an average claim score of 89.77 versus 75.40 for MCP‑Zero, and it lies on the Pareto frontier for task performance and input‑token cost.
OmicsBench is a new long‑horizon, multi‑turn bioinformatics benchmark for AI agents, comprising 97 tasks that link 662 subtasks into directed acyclic graphs. Each subtask is a user turn scored deterministically, with optional in‑context feedback. Across 12 models, agents achieve mean scores between 0.43 and 0.67, revealing lower performance on dependent subtasks and highlighting qualitative failure modes through an unsupervised discovery pipeline.
arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.
arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.
arXiv:2409. 11363v2 Announce Type: replace-cross Abstract: AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research.
arXiv:2606. 02060v1 Announce Type: new Abstract: Deep-research agents solve tasks through long trajectories of search, tool use, evidence inspection, and answer synthesis.
GRADS is a graph-guided framework for generating synthetic data that explicitly controls difficulty by maintaining a graph of evidence and inference dependencies. It iteratively selects seeds, expands evidence, constructs question–answer pairs, and verifies them, ensuring that each generated task contains only indispensable inference steps. Experiments across security, software engineering, and scientific domains show that GRADS produces verified tasks with distinct difficulty levels, and training models on this data yields significant performance gains, including high solution-termination rates and reduced tool usage.
arXiv:2606. 12736v1 Announce Type: new Abstract: AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood.
Agentick is a unified benchmark for sequential decision‑making agents that evaluates RL, LLM, VLM, hybrid, and human agents on 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities via a single Gymnasium‑compatible interface. It includes a Coding API, oracle reference policies, pre‑built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation of 27 configurations and over 90,000 episodes shows no single approach dominates, with GPT‑5 mini leading overall, PPO excelling in planning and multi‑agent tasks, and the reasoning harness boosting LLM performance by 3‑10×, while ASCII observations outperform natural language.