arXiv Computer Vision
GUI-Lens: Enabling Precise GUI Grounding for Off-the-Shelf VLMs via Coarse-to-Fine Cropping
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv AI
Aug 5
GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
arXiv:2608. 03270v1 Announce Type: cross Abstract: GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents.
arXiv Computation and Language
Oct 27
PreView: Enhancing GUI Grounding with Preprocessed Views and Cues
arXiv:2610.33645v1 Announce Type: cross Abstract: GUI grounding requires locating small controls in cluttered screenshots and identifying which control a natural-language instruction refers to. Cropp...
Hugging Face Trending Papers
Oct 13
MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search
The paper introduces MHVF, a training‑free framework for GUI grounding that uses multi‑hypothesis exploration and local visual verification with a single pretrained vision‑language model. MHVF generates spatial hypotheses from full‑screen predictions, refines them through zoomed re‑localization, and verifies candidate agreement with the instruction to resolve ambiguity. Experiments on eight benchmarks with five models show significant accuracy gains, notably improving UI‑Venus‑Ground‑7B by 4.3 pp on Mind2Web and raising Qwen2.5‑VL‑7B’s ScreenSpot‑Pro accuracy from 23.6 % to 53.6 %.
arXiv Computation and Language
Sep 17
RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
RankGround is a two‑stage framework for GUI grounding that uses a single Vision‑Language Model call per query. It introduces GroundRanker, a lightweight multimodal reranker that selects the most promising crop from a dense candidate set, trained with a two‑stage curriculum on ranking supervision data derived from existing grounding datasets. Experiments show RankGround outperforms strong baselines, achieving 1.4× faster inference and a 5.5% average improvement in localization accuracy over the second‑best method across all backbones and screen scales.
arXiv AI
Jul 1
GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding
arXiv:2511. 00810v4 Announce Type: replace-cross Abstract: Graphical user interface (GUI) grounding is a key capability for computer-use agents, mapping natural-language instructions to actionable regions on the screen.
Hugging Face Trending Papers
Aug 10
Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching
GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates.