AdaGround is a training‑free framework for GUI grounding that addresses shortcomings of existing zooming methods. It introduces Token‑Normalized Adaptive Zooming to match magnification to target size, a Verifier‑in‑the‑Loop Refinement that uses a simple Yes/No click to accept or reject zooms and decide when to abstain, and Compositional Structure Grounding for spreadsheet cells. Experiments on Qwen3‑VL models show AdaGround outperforms state‑of‑the‑art training‑free zooming techniques and many trained GUI models, boosting accuracy by 7.4 to 20.6 points on OSWorld‑G and ScreenSpot‑Pro and maintaining gains on unseen UI‑Vision data.
arXiv:2610.33645v1 Announce Type: cross Abstract: GUI grounding requires locating small controls in cluttered screenshots and identifying which control a natural-language instruction refers to. Cropp...
arXiv:2608. 03270v1 Announce Type: cross Abstract: GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents.
The paper introduces a contrastive curriculum for teaching large language models to use a zoom‑in tool without supervised fine‑tuning, leveraging an InfoNCE‑style intrinsic reward that progressively hardens negative examples. Experiments on V*, HRBench, and MME‑RealWorld demonstrate competitive performance and efficiency, with the method even surpassing baselines when replacing traditional SFT. A new synthetic dataset, Muffin&Chihuahua (M&C), is created to directly evaluate zoom‑in capability, revealing that recall best correlates with task success.
GUI‑Lens is a coarse‑to‑fine grounding framework that lets a general‑purpose vision‑language model (VLM) iteratively crop and zoom into a GUI screenshot to locate a target element precisely. By extracting OCR text and UI component positions as coordinate references, the VLM selects successive regions to focus on, refining its view until the target is identified. Experiments on five GUI grounding benchmarks and six base models show that GUI‑Lens consistently improves grounding accuracy, achieving up to 9.3× the accuracy of direct grounding on ScreenSpot‑Pro.
arXiv:2604. 14262v2 Announce Type: replace-cross Abstract: GUI grounding models report over 85% accuracy on standard benchmarks, yet drop 27-56 percentage points when instructions require spatial reasoning rather than direct element naming.
arXiv:2609.06490v1 Announce Type: cross Abstract: Recursive Super-Resolution (SR) extends fixed-scale SR to extreme magnification by repeatedly feeding predictions back into the same model, analogous...
arXiv:2511. 00810v4 Announce Type: replace-cross Abstract: Graphical user interface (GUI) grounding is a key capability for computer-use agents, mapping natural-language instructions to actionable regions on the screen.
RankGround is a two‑stage framework for GUI grounding that uses a single Vision‑Language Model call per query. It introduces GroundRanker, a lightweight multimodal reranker that selects the most promising crop from a dense candidate set, trained with a two‑stage curriculum on ranking supervision data derived from existing grounding datasets. Experiments show RankGround outperforms strong baselines, achieving 1.4× faster inference and a 5.5% average improvement in localization accuracy over the second‑best method across all backbones and screen scales.
arXiv:2610.19624v1 Announce Type: new Abstract: Visual recognition models usually process a fixed view under a fixed computation budget, although images often benefit from different scales and amount...
IVSGround introduces a lightweight view selector that learns to choose the most informative camera views for vision‑language model (VLM) based 3D visual grounding, replacing heuristic view selection. The selector is trained via a two‑stage rejection sampling process that uses feedback from a reasoning VLM to generate supervision signals. Experiments on ScanRefer and NR3D demonstrate that IVSGround consistently improves grounding accuracy over existing zero‑shot pipelines, underscoring the importance of selecting where to look for effective 3D visual grounding.
arXiv:2609.23733v1 Announce Type: new Abstract: Feed-forward visual geometry models such as the Visual Geometry Grounded Transformer (VGGT) have recently enabled direct 3D reconstruction from multi-v...
The paper introduces MHVF, a training‑free framework for GUI grounding that uses multi‑hypothesis exploration and local visual verification with a single pretrained vision‑language model. MHVF generates spatial hypotheses from full‑screen predictions, refines them through zoomed re‑localization, and verifies candidate agreement with the instruction to resolve ambiguity. Experiments on eight benchmarks with five models show significant accuracy gains, notably improving UI‑Venus‑Ground‑7B by 4.3 pp on Mind2Web and raising Qwen2.5‑VL‑7B’s ScreenSpot‑Pro accuracy from 23.6 % to 53.6 %.