← Back to all news
arXiv AI By Bingchao Wang, Yongcheng Jing, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao

AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding

Read the original on arXiv AI →

AdaGround is a training‑free framework for GUI grounding that addresses shortcomings of existing zooming methods. It introduces Token‑Normalized Adaptive Zooming to match magnification to target size, a Verifier‑in‑the‑Loop Refinement that uses a simple Yes/No click to accept or reject zooms and decide when to abstain, and Compositional Structure Grounding for spreadsheet cells. Experiments on Qwen3‑VL models show AdaGround outperforms state‑of‑the‑art training‑free zooming techniques and many trained GUI models, boosting accuracy by 7.4 to 20.6 points on OSWorld‑G and ScreenSpot‑Pro and maintaining gains on unseen UI‑Vision data.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Sep 4

Learning to Zoom Efficiently with a Contrastive Curriculum

The paper introduces a contrastive curriculum for teaching large language models to use a zoom‑in tool without supervised fine‑tuning, leveraging an InfoNCE‑style intrinsic reward that progressively hardens negative examples. Experiments on V*, HRBench, and MME‑RealWorld demonstrate competitive performance and efficiency, with the method even surpassing baselines when replacing traditional SFT. A new synthetic dataset, Muffin&Chihuahua (M&C), is created to directly evaluate zoom‑in capability, revealing that recall best correlates with task success.

By Falko Helm, Iryna Gurevych
agentsfine-tuning
More like this →
arXiv Computer Vision
5d ago

GUI-Lens: Enabling Precise GUI Grounding for Off-the-Shelf VLMs via Coarse-to-Fine Cropping

GUI‑Lens is a coarse‑to‑fine grounding framework that lets a general‑purpose vision‑language model (VLM) iteratively crop and zoom into a GUI screenshot to locate a target element precisely. By extracting OCR text and UI component positions as coordinate references, the VLM selects successive regions to focus on, refining its view until the target is identified. Experiments on five GUI grounding benchmarks and six base models show that GUI‑Lens consistently improves grounding accuracy, achieving up to 9.3× the accuracy of direct grounding on ScreenSpot‑Pro.

By Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li, Yimin Deng, Jingtong Gao, Junjia Qi, Hanyu Yan, Yefeng Zheng, Xiaopeng Li, Wanyu Wang, Xian Wu, Xiangyu Zhao
llmsagentscomputer-visionmultimodalbenchmarks
More like this →