The paper introduces a label‑free test‑time training approach for GUI grounding, leveraging confidence patterns in coordinate tokens rather than full‑sequence confidence. It proposes Confidence‑Anchored Learning (CAL) to filter pseudo‑labels and assign distance‑based binary rewards, and extends this to Confidence‑Anchored Negative Learning (CANL) which optimizes solely on negative samples to avoid noisy positives. Experiments show that CANL‑7B achieves 92.1% on ScreenSpot‑V2 and 33.8% on ScreenSpot‑Pro, improving the base model by 8.9%.
arXiv:2606. 29705v1 Announce Type: new Abstract: Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models.
arXiv:2511. 07332v2 Announce Type: replace-cross Abstract: Building reliable computer-use agents requires grounding: accurately connecting natural language instructions to the correct on-screen elements.
arXiv:2609.39547v1 Announce Type: new Abstract: GUI agents built on large language and vision-language models still struggle on unseen applications and complex multi-step tasks, as completing real GU...
arXiv:2606. 24515v1 Announce Type: new Abstract: Computer-Use Agents (CUAs) execute high-level user goals by perceiving and acting directly within graphical user interfaces.
arXiv:2608. 11191v1 Announce Type: cross Abstract: GUI Visual Grounding is a fundamental capability for GUI agents.
arXiv:2606. 03608v1 Announce Type: cross Abstract: Test-time reinforcement learning has emerged as a promising paradigm for enhancing the complex reasoning abilities of large language models in a completely label-free manner.
PointRL introduces a verifiable reinforcement learning framework that learns point-level vision‑language grounding from heterogeneous annotation evidence such as bounding boxes, masks, and instance labels. The method converts these annotations into pointing instructions while preserving target supports, instance membership, and set constraints as hidden verifier evidence, which a deterministic checker uses to score predictions. Evaluation on PointArena shows that PointRL improves Qwen3.5‑4B’s accuracy from 56.11% to 65.58%, and similar gains are observed on RoboSpatial, BLINK, and Ref‑Adv benchmarks.
arXiv:2606. 18101v1 Announce Type: new Abstract: Graphical user interface (GUI) grounding requires vision-language models (VLMs) to identify small target elements in high-resolution screenshots and predict precise screen coordinates.
arXiv:2606. 00083v1 Announce Type: cross Abstract: Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics.
arXiv:2606. 11078v1 Announce Type: new Abstract: Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments.
arXiv:2609.36939v1 Announce Type: new Abstract: GUI agents automate tasks on digital devices by grounding language instructions in visual interfaces. Existing group-relative reinforcement learning im...
arXiv:2607. 10891v1 Announce Type: new Abstract: Large language models (LLMs) are rapidly shifting toward agents that solve tasks through diverse interfaces, including web and graphical user interfaces (GUIs).