ScreenSearch is a system for exploring desktop GUI states under partial observability. It combines structural screen retrieval, deduplication, and an ambiguity-aware PUCT graph-bandit to expand the reachable frontier while reducing state ambiguity. Across 11 applications, it collected over 1 M screenshots and 30 K deduplicated states, demonstrating that both ambiguity reduction and frontier exploration are essential for effective corpus building.
arXiv:2609.39547v1 Announce Type: new Abstract: GUI agents built on large language and vision-language models still struggle on unseen applications and complex multi-step tasks, as completing real GU...
arXiv:2609.10297v1 Announce Type: new Abstract: GUI agents accumulate high-resolution screenshots as the trajectory unfolds, increasing inference latency and memory usage. Training-free visual token...
arXiv:2608. 12847v1 Announce Type: new Abstract: Retrieval can identify a past trajectory that may matter, yet it does not specify how an acting agent should use that trajectory after users, entities, constraints, or environment state have changed.
OS-Marathon is a new benchmark that tests computer‑use agents on vast‑horizon, repetitive tasks, covering 100 tasks across five scenarios and ten domains. The study shows that current state‑of‑the‑art agents perform poorly on these tasks, and that simply decomposing workflows into subtasks does not solve the problem. Introducing a cost‑friendly personalization method called GraphDemo, which adapts agents from a single human demonstration, improves performance, highlighting the value of human guidance for these challenging tasks.
arXiv:2607. 26041v1 Announce Type: new Abstract: Computer-use agents (CUAs) increasingly act through desktop GUIs to complete long-horizon tasks.
arXiv:2607. 13884v1 Announce Type: new Abstract: Large Language Model (LLM) agents have shown remarkable capabilities in autonomous decision-making by generating sequential trajectories of states, actions, and observations.
arXiv:2608. 15930v1 Announce Type: new Abstract: Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution.
Terminal-Universe is a framework that converts large collections of terminal-based agent trajectories into reusable, executable environments. By replaying recorded file operations and filling missing files with a completion agent, it reconstructs the original workspace and generates new tasks, scaling them in breadth (cross-workspace queries) and depth (multi-round interactions). The resulting 37.3k task-sufficient environments enable significant performance gains when fine‑tuning language models on terminal‑centric benchmarks.
The paper introduces an online skill‑evolution framework that transforms interaction traces and evaluator feedback into a persistent, versioned library of reusable procedures for computer‑use agents. By executing each iteration against a frozen library snapshot, the system updates skills without altering the underlying model parameters. Experiments across four OSWorld domains show that the evolving library consistently outperforms an empty‑library baseline, with gains ranging from 5.7 to 18.6 percentage points, while also revealing domain‑specific temporal stability and challenges in skill retrieval and revision.
arXiv:2608. 02352v1 Announce Type: new Abstract: Native computer use offers a general interface for agents to operate almost any software available to people, but requires long-horizon state tracking, large-scale interactive experience, and learning from sparse yet verifiable outcomes.
arXiv:2607. 23263v1 Announce Type: new Abstract: Deciding whether a trajectory actually fulfills its instruction governs how we measure computer-use agents on long-horizon graphical-user-interface tasks and how we train them with reinforcement learning.
arXiv:2606. 11662v1 Announce Type: new Abstract: Deep search requires agents to answer complex questions through multi-step web search, browsing, evidence comparison, and synthesis.