arXiv AI
Matched Capabilities, Different Behaviors: A Controlled Study of MCP and CLI Interfaces for Language Agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 7
Matching Matters: A Fair Quality-Efficiency Benchmark for Command-Line Agents
arXiv:2606. 21140v2 Announce Type: replace-cross Abstract: Rapid advances in large language models have improved the task-solving capabilities of command-line-interface (CLI)-based agents, whose CLIs determine how models invoke tools, maintain interaction history, and recover from failures.
arXiv AI
Sep 3
CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI
CivBench is an open‑source benchmark that evaluates language‑model agents in the long‑horizon, tool‑mediated game Civilization VI using the Model Context Protocol (MCP). Each episode lasts over 300 turns, generating thousands of tool calls across a 76‑tool action space, and includes a narration layer that translates visual game state into structured text. The study characterises agent behaviour across four model families, introducing Proactive Monitoring Rate (PMR) and RAG@10 as interface‑level metrics, and finds that agents often under‑monitor strategic state and fail to execute near‑term commitments despite tool access and explicit guidance.
arXiv AI
Jul 14
AgentAbstain: Do LLM Agents Know When Not to Act?
arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.
arXiv AI
Aug 24
Terminal Agents: A Survey of AI Agents in Command-Line Environments
The paper surveys AI agents that operate primarily through command-line terminals, defining them as systems whose main action loop involves executing terminal commands, receiving textual feedback, and interacting with a stateful environment. It introduces a seven‑dimensional terminal competence profile to link system architecture, learning, and evaluation, and highlights how behavior is jointly shaped by the model, interface, harness, runtime, and environment. The authors argue for explicit reporting of system and runtime conditions, supported by replayable traces and process‑level evidence, to better understand and benchmark terminal‑mediated agency.
arXiv AI
Sep 18
An Empirical Study of Harness Design for Coding Agents
The study investigates how individual components of a coding harness—planning, action space, and context management—affect autonomous coding agents’ performance. By fixing the execution loop and varying these components across 176 settings on SWE‑Bench Verified and Terminal‑Bench 2.1, the authors find that context management is most valuable when context windows are tight, staging rule‑based elision before LLM summarization yields the best efficiency, planning serves as an accuracy scaffold for weaker models and a cost saver for stronger ones, and predefined tools help models with limited bash skills while bash‑capable models benefit from a bash‑only interface. Trajectory‑level analysis shows that context management lengthens execution paths, planning alters where trajectories terminate, and the action space determines code granularity, offering a modular framework for future harness design.
arXiv AI
Jun 24
GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents
arXiv:2606. 24551v1 Announce Type: new Abstract: Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions.