RealCLI is a new benchmark for evaluating human‑agent collaboration in command‑line interface (CLI) tasks. It unifies task execution and collaboration quality assessment through a five‑dimensional framework with 22 sub‑dimensions and 51 interaction templates. The benchmark offers two modes—REALCLI‑EXEC with full specifications and REALCLI‑COLLAB with a scripted user providing progressive feedback—and reveals significant capability gaps in current agents, especially in steerability, trust calibration, and safety management.
arXiv:2607. 13056v1 Announce Type: cross Abstract: Current vision-language-action (VLA) benchmarks primarily evaluate isolated manipulation skills while leaving human-robot interaction structure largely unmodeled.
arXiv:2606. 09426v1 Announce Type: new Abstract: Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools.
Kolab is a new dataset comprising over 115 hours of screen recordings from 50 professionals across 40 occupations who collaborate with AI agents on realistic, long‑horizon tasks. The dataset includes participants’ judgments of collaboration quality and output, revealing varied coordination patterns and the need for agents to anticipate how experts will use or modify their contributions. Analysis shows current state‑of‑the‑art models overestimate collaboration success and struggle to differentiate high‑ from low‑quality interactions, indicating a gap in understanding long‑horizon human‑agent teamwork.
arXiv:2606. 21140v2 Announce Type: replace-cross Abstract: Rapid advances in large language models have improved the task-solving capabilities of command-line-interface (CLI)-based agents, whose CLIs determine how models invoke tools, maintain interaction history, and recover from failures.
arXiv:2606. 03103v1 Announce Type: new Abstract: Real-world professional desktop workflows in specialized creative and engineering software unfold over long horizons and often require human-in-the-loop coordination, where agents proactively seek necessary information and users provide additional instructions, clarifications, feedback, or corrections as the task progresses.
CoworkOS is a general‑purpose harness designed for long‑horizon real‑world automation, separating task strategy from execution management. Its three‑layer architecture allows direct execution or delegation to subagents, while a shared application layer lets humans and agents collaboratively edit documents, designs, and 3D scenes. The system incorporates hierarchical workspaces and persistent semantic memory to maintain continuity across agents and sessions, and demonstrates strong performance across 12 benchmarks and six application categories, outperforming leading agents like Codex and DeepSeek Harness.
arXiv:2606. 28480v1 Announce Type: cross Abstract: As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding.
arXiv:2604. 13072v2 Announce Type: replace-cross Abstract: OpenClaw-style personal assistants extend LLM agents from isolated tool use to open-ended, stateful, and personalized software environments.
arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.
RealClawBench is a live benchmark framework derived from real OpenClaw developer‑agent sessions, designed to capture the distribution, diversity, and real‑world difficulty of deployed agent use. It reconstructs execution environments and uses deterministic verifiable scorers to convert real sessions into reproducible, automatically scored tasks, yielding 281 executable tasks with minimal distribution shift. Evaluation of 14 contemporary models shows the best system solves only 65.8% of tasks, highlighting significant room for improvement on realistic workloads.
arXiv:2605.22535v2 Announce Type: replace Abstract: We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal...
arXiv:2606. 24551v1 Announce Type: new Abstract: Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions.