arXiv AI
How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computer Vision
Sep 15
Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding
The paper introduces a label‑free test‑time training approach for GUI grounding, leveraging confidence patterns in coordinate tokens rather than full‑sequence confidence. It proposes Confidence‑Anchored Learning (CAL) to filter pseudo‑labels and assign distance‑based binary rewards, and extends this to Confidence‑Anchored Negative Learning (CANL) which optimizes solely on negative samples to avoid noisy positives. Experiments show that CANL‑7B achieves 92.1% on ScreenSpot‑V2 and 33.8% on ScreenSpot‑Pro, improving the base model by 8.9%.
arXiv Computer Vision
Oct 12
ConClick: Element-Wise Contrastive Learning Improves GUI Grounding
ConClick introduces a contrastive learning framework that improves GUI grounding by teaching models to distinguish target interface elements from plausible distractors. The authors built ConClick-Data, a dataset of 4,292 challenging target-distractor pairs, and trained the model with listwise ranking and pairwise margin objectives, achieving significant accuracy gains on multiple benchmarks. An inference-time strategy called ReClick further boosts performance by revisiting promising regions and comparing competing predictions.
Hugging Face Trending Papers
Oct 13
MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search
The paper introduces MHVF, a training‑free framework for GUI grounding that uses multi‑hypothesis exploration and local visual verification with a single pretrained vision‑language model. MHVF generates spatial hypotheses from full‑screen predictions, refines them through zoomed re‑localization, and verifies candidate agreement with the instruction to resolve ambiguity. Experiments on eight benchmarks with five models show significant accuracy gains, notably improving UI‑Venus‑Ground‑7B by 4.3 pp on Mind2Web and raising Qwen2.5‑VL‑7B’s ScreenSpot‑Pro accuracy from 23.6 % to 53.6 %.
arXiv Computation and Language
Aug 25
GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
arXiv:2608.21832v1 Announce Type: new Abstract: Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether mo...
arXiv AI
Jun 30
GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots
arXiv:2606. 29705v1 Announce Type: new Abstract: Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models.
arXiv Computer Vision
Oct 15
STP-GUI: Spatially-Aware Token Pruning for Efficient Web-GUI Agents
arXiv:2610.20390v1 Announce Type: new Abstract: With the rapid development of vision-language models (VLMs), Web-GUI agents have achieved promising performance in interacting with real-world interfac...