← Back to all news
arXiv AI By Ruilin Yao, Xinyu Qiu, Tianyu Zou, Yi Rong, Shengwu Xiong, Shuheng Shen, Changhua Meng

Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding

Read the original on arXiv AI →

The paper introduces LBS (Look Beyond Screens), a vision‑language model that learns transferable visual observation behaviors from GUI grounding. LBS is first supervised with teacher‑generated GUI trajectories and then fine‑tuned via reinforcement learning to make multi‑turn crop‑and‑inspect decisions. Experiments show that LBS models based on Qwen3.5‑35B‑A3B and Qwen3.8‑27B outperform their baselines on PerceptionBench, HR‑Bench, and ChartQAPro, indicating that the learned observation strategies generalize beyond GUI‑specific imitation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Oct 15

GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

arXiv:2610.19100v1 Announce Type: cross Abstract: Constructing robust GUI agents relies on multimodal visual grounding: the ability to precisely map natural language intents to actionable on-screen c...

By Qingsong Xie, Jiahao Tian, Jiacheng Tang, Zhifeng Li, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das
agentsreinforcement-learningfine-tuningefficiencymultimodalbenchmarks
More like this →
Hugging Face Trending Papers
Oct 13

MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search

The paper introduces MHVF, a training‑free framework for GUI grounding that uses multi‑hypothesis exploration and local visual verification with a single pretrained vision‑language model. MHVF generates spatial hypotheses from full‑screen predictions, refines them through zoomed re‑localization, and verifies candidate agreement with the instruction to resolve ambiguity. Experiments on eight benchmarks with five models show significant accuracy gains, notably improving UI‑Venus‑Ground‑7B by 4.3 pp on Mind2Web and raising Qwen2.5‑VL‑7B’s ScreenSpot‑Pro accuracy from 23.6 % to 53.6 %.

llmsagentsmultimodalbenchmarks
More like this →
arXiv Computer Vision
Aug 27

PointRL: Learning Point-Level Vision-Language Grounding from Verifiable Annotation Evidence

PointRL introduces a verifiable reinforcement learning framework that learns point-level vision‑language grounding from heterogeneous annotation evidence such as bounding boxes, masks, and instance labels. The method converts these annotations into pointing instructions while preserving target supports, instance membership, and set constraints as hidden verifier evidence, which a deterministic checker uses to score predictions. Evaluation on PointArena shows that PointRL improves Qwen3.5‑4B’s accuracy from 56.11% to 65.58%, and similar gains are observed on RoboSpatial, BLINK, and Ref‑Adv benchmarks.

By Jingyang Su, Pu Cao, Xiuze Jin, Longyue Zhang, Qing Song, Lu Yang
llmsreinforcement-learningroboticsmultimodalbenchmarks
More like this →