arXiv AI
OS-Wizard: Scaling Computer-Using Agents to Specialized Software via Knowledge-Guided Exploration
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 23
ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents
ToolCUA is an end‑to‑end agent that learns to optimally orchestrate GUI actions and tool calls for Computer Use Agents. It introduces a staged training pipeline that first scales interleaved GUI‑tool trajectories from static GUI data, then bootstraps decision making with single‑turn reinforcement learning, and finally refines performance via online agentic RL guided by a tool‑efficient path reward. On the OSWorld‑MCP benchmark, ToolCUA achieves 46.85% accuracy, a 66% relative improvement over the baseline and a 3.9% gain over GUI‑only models, setting a new state of the art for comparable‑scale models.
Hugging Face Trending Papers
Sep 29
HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents
Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical user interface (GUI) interactions, which are often...
arXiv Computer Vision
Sep 30
HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents
arXiv:2609.38008v1 Announce Type: new Abstract: Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical us...
arXiv AI
Oct 19
Adaptation of Computer Use Agents to New Software with User Manuals
The paper introduces MAGE, a framework that uses user manuals to enhance computer use agents (CUAs) for new software. MAGE retrieves relevant manual content during execution, generates procedural guidance, and adapts the agent policy through interaction. In training, it converts manuals into executable task-plan pairs to guide rollouts, reducing the need for human‑annotated demonstrations and improving task success rates across six applications.
arXiv AI
Jul 13
Improving Language Agents through BREW: Bootstrapping expeRientially-learned Environmental knoWledge
arXiv:2511. 20297v2 Announce Type: replace Abstract: Large Language Model (LLM)-based agents are increasingly capable of complex, multi-step tasks such as GUI automation, tool use, and data manipulation, yet they cannot learn from experience: each new session rediscovers solutions from scratch.
arXiv AI
Oct 16
OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents
OS-Helix is an end‑to‑end multimodal agent that unifies graphical user interface (GUI) actions and code execution into a single action space. It is trained on high‑quality hybrid data generated through application‑artifact composition and a decoupled instruction/evaluator pipeline, and further refined with online reinforcement learning using a step‑balanced GSPO algorithm. The resulting models achieve significant performance gains over hybrid baselines on the OSWorld‑Verified benchmark and demonstrate cross‑platform transfer to WindowsAgentArena.