arXiv AI
CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
1d ago
Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives
The paper introduces Tool Primitives, a design that replaces rigid API schemas with natural language interfaces for tool calling, enabling seamless inter-tool communication. It builds ToolFace, a repository of over 25,000 functions that LLMs can dynamically retrieve, and HEART, a harness engineering framework that orchestrates tool use with planning, routing, and verification. Experiments show HEART outperforms fine‑tuned models and leading commercial LLMs while cutting API costs by up to 85%.
arXiv AI
Jul 24
DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers
arXiv:2607. 20531v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly deployed over Model Context Protocol (MCP) servers, yet the benchmarks used to evaluate them score the final answer or a fixed "ground-truth" list of tools, both of which are fragile once the underlying data is live and stateful.
arXiv AI
Jul 15
Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution
arXiv:2607. 13034v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly automate multi-step engineering and informatics workflows, yet they rarely ask how much effort a task actually requires.
arXiv AI
Jun 30
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents
arXiv:2511. 02734v3 Announce Type: replace Abstract: Current evaluations of Large Language Model (LLM) agents primarily emphasize task completion, often overlooking resource efficiency and adaptability.
arXiv AI
Aug 13
VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies
arXiv:2608. 12282v1 Announce Type: new Abstract: Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation.
arXiv AI
Jul 14
AgentAbstain: Do LLM Agents Know When Not to Act?
arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.