The paper introduces LBS (Look Beyond Screens), a vision‑language model that learns transferable visual observation behaviors from GUI grounding. LBS is first supervised with teacher‑generated GUI trajectories and then fine‑tuned via reinforcement learning to make multi‑turn crop‑and‑inspect decisions. Experiments show that LBS models based on Qwen3.5‑35B‑A3B and Qwen3.8‑27B outperform their baselines on PerceptionBench, HR‑Bench, and ChartQAPro, indicating that the learned observation strategies generalize beyond GUI‑specific imitation.
arXiv:2610.19100v1 Announce Type: cross Abstract: Constructing robust GUI agents relies on multimodal visual grounding: the ability to precisely map natural language intents to actionable on-screen c...
arXiv:2610.02320v1 Announce Type: cross Abstract: Computer-use agents need to reliably ground action targets in complex desktop scenes, where multiple applications, overlapping windows, and visually...
The paper introduces MHVF, a training‑free framework for GUI grounding that uses multi‑hypothesis exploration and local visual verification with a single pretrained vision‑language model. MHVF generates spatial hypotheses from full‑screen predictions, refines them through zoomed re‑localization, and verifies candidate agreement with the instruction to resolve ambiguity. Experiments on eight benchmarks with five models show significant accuracy gains, notably improving UI‑Venus‑Ground‑7B by 4.3 pp on Mind2Web and raising Qwen2.5‑VL‑7B’s ScreenSpot‑Pro accuracy from 23.6 % to 53.6 %.
PointRL introduces a verifiable reinforcement learning framework that learns point-level vision‑language grounding from heterogeneous annotation evidence such as bounding boxes, masks, and instance labels. The method converts these annotations into pointing instructions while preserving target supports, instance membership, and set constraints as hidden verifier evidence, which a deterministic checker uses to score predictions. Evaluation on PointArena shows that PointRL improves Qwen3.5‑4B’s accuracy from 56.11% to 65.58%, and similar gains are observed on RoboSpatial, BLINK, and Ref‑Adv benchmarks.
arXiv:2606. 16122v1 Announce Type: new Abstract: Visual thinking should not only sound right; it should show its evidence.
arXiv:2608. 11191v1 Announce Type: cross Abstract: GUI Visual Grounding is a fundamental capability for GUI agents.
Think3D introduces a framework that endows Vision‑Language Models with interactive 3D chain‑of‑thought reasoning by integrating 3D manipulation tools for active spatial exploration. The approach improves performance on benchmarks such as BLINK Multi‑view, MindCube‑1K, and VSI‑Bench‑Tiny for proprietary models like GPT‑4.1 and Gemini 2.5 Pro, and a reinforcement‑learning variant, Think3D‑RL, enables open‑weight models such as Qwen3‑VL‑4B to autonomously learn effective 3D exploration strategies, yielding tool‑use patterns comparable to stronger models and turning a performance drop on MindCube‑1K into a substantial improvement.
The paper introduces ACT3D, a vision‑language model that learns to think and explore in 3D environments using only single‑image inputs and action‑based interactions. By fine‑tuning with trajectory supervision and then applying reinforcement learning driven by final‑answer correctness, ACT3D develops an exploration policy that adapts to the question and improves accuracy across six benchmarks, outperforming both proprietary and open‑source baselines while using fewer input frames. The study shows that outcome‑based RL can teach VLMs to decide when and how to acquire visual evidence, leading to better reasoning and spatial understanding.
GUIFlow is a framework that enables temporal redundancy‑aware incremental inference for GUI agents built on vision‑language models. By selectively updating computational states—such as position‑aligned KV states, vision encoder outputs, and language model drafts—across interaction steps, it allows cross‑step reuse of visual encoding, prefill, and decoding without extra training or model changes. Experiments show up to 3.39× speedup on offline benchmarks and 3.49× in online OSWorld interactions while preserving task performance.
The paper introduces GTA‑VLA, an interactive Vision‑Language‑Action framework that lets users guide robot policies with explicit visual cues such as affordance points, boxes, and traces. Unlike traditional direct sense‑to‑act models, GTA‑VLA incorporates a spatial‑visual Chain‑of‑Thought that blends human guidance with internal task planning, and couples this reasoning module with a lightweight reactive action head for efficient execution. Experiments on the SimplerEnv WidowX benchmark show a state‑of‑the‑art 81.2 % success rate, and the framework significantly improves task success under out‑of‑domain visual shifts and spatial ambiguities, demonstrating the benefit of interactive reasoning for failure recovery in embodied control.
arXiv:2606. 29705v1 Announce Type: new Abstract: Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models.
arXiv:2510. 14828v3 Announce Type: replace Abstract: Improving the reasoning capabilities of embodied agents is crucial for robots to complete complex human instructions in long-view manipulation tasks successfully.