← Back to all news
arXiv AI By Kyeongseon Kim, Jiyeon Son, Tae-Hyun Oh

IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints

Read the original on arXiv AI →

IGG is a new benchmark for Interactive GUI Grounding that requires agents to first interact with a GUI to make a target visible and then click it. It includes 601 tasks across 16 controllable environments and seven visibility-constraint sub‑types, with a compact action space of scrolling, moving, dragging, waiting, and clicking. Human participants achieve 94.6% accuracy, while GPT‑5.5 and the best open‑source VLM reach 79.4% and 31.6% respectively, highlighting a gap mainly due to the difficulty of executing reveal interactions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Aug 18

UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

arXiv:2608. 15930v1 Announce Type: new Abstract: Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution.

By Zihan Ding, Longxu Dou, Qi Gao, Xiangwu Guo, Shengchao Hu, Zilong Huang, Zihang Jiang, Lei Ke, Mengcheng Lan, Weixian Lei, Hanxuan Li, Honglin Li, Xiyun Li, Zaitang Li, Leowei Liang, Xin Luo, Haozhe Ma, Jiayi Mao, Zhoujie Pan, Can Qin, Tianyuan Qu, Weiqi Wang, Wenkai Wang, Yonglin Wang, Yuxin Wang, Chenxu Wu, Yingchen Yu, Chenyu Zhang, Yuhao Zheng
agentsreinforcement-learningfine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Sep 2

GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments

GUI-CC is a benchmark designed to assess the contextual consistency of GUI world models when used as agent environments, rather than just one‑step next‑screen predictors. It includes two tracks: an offline reference‑action track that rolls models along real mobile GUI trajectories, and an online agent‑loop track where fixed probing agents interact with model‑generated UIs. The benchmark evaluates transition fidelity, plausibility, contextual consistency, and task progress across 500 offline trajectory tasks and 200 online tasks spanning 30 mobile apps.

By Lin Fu, Zheyuan Yang, Tianhui Zhang, Jinbiao Wei, Guo Gan, Boxu Liu, Yilun Zhao, Yu Rong
agentsbenchmarks
More like this →
arXiv AI
Jun 10

A History-Aware Visually Grounded Critic for Computer Use Agents

arXiv:2606. 11078v1 Announce Type: new Abstract: Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments.

By Jaewoo Lee, Zaid Khan, Archiki Prasad, Justin Chih-Yao Chen, Supriyo Chakraborty, Kartik Balasubramaniam, Sambit Sahu, Elias Stengel-Eskin, Hyunji Lee, Mohit Bansal
llmsagentsmultimodalbenchmarks
More like this →