arXiv AI
Jul 8
UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
arXiv:2607. 06306v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated growing competence in web page generation.
UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence.
Related stories
arXiv AI
Jun 12
VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents
arXiv:2605. 26144v2 Announce Type: replace-cross Abstract: We present VISTA (VIsual Spec-To-App Benchmark), a benchmark for evaluating the end-to-end web-app generation capabilities of LLM-based agents.
arXiv AI
Jun 2
Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages
arXiv:2606. 00154v1 Announce Type: cross Abstract: Recent advancements in multimodal large language models (MLLMs) have achieved remarkable progress in multimodal reasoning and code generation, catalyzing a new paradigm for front-end development.
arXiv AI
Aug 11
Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents
arXiv:2508. 04412v3 Announce Type: replace Abstract: The advent of large language models (LLMs) has sparked an evolution of autonomous web browsing agents: given a web browsing task and serialised user interface (UI) state, an LLM is expected to suggest input actions that incrementally solve the given task.
arXiv AI
Jul 14
MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation
arXiv:2607. 10079v1 Announce Type: new Abstract: Digital Adoption Platforms (DAPs) are embedded overlays widely used on web systems to guide users through operations inside a page, helping them get started with unfamiliar interfaces quickly.
arXiv AI
Aug 3
Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation
arXiv:2607. 28645v1 Announce Type: cross Abstract: Recent multimodal large language models can convert visual designs directly into executable code, but real mobile products require multiple screenshots to become a buildable codebase with shared components and working navigation.
arXiv AI
Aug 7
AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents
arXiv:2608. 05891v1 Announce Type: new Abstract: Mobile GUI agents can operate apps through pixel perception and touch actions, making them a promising interface for collecting and improving long-horizon mobile interaction policies.
arXiv AI
Jun 17
LongWebBench: Evaluating Structural and Functional Webpage Generation in Long-Horizon Settings
arXiv:2606. 17727v1 Announce Type: new Abstract: Recent vision-language models (VLMs) have shown promising progress in generating webpages from visual inputs, yet existing evaluations mainly focus on short, single-screen, and largely static webpages.
arXiv AI
Jun 12
Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach
arXiv:2606. 13192v1 Announce Type: new Abstract: User experience (UX) centered on usability, perceived consistency, and functional clarity is fundamental to real-world user interfaces (UI).
arXiv AI
Aug 11
CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception
arXiv:2608. 08392v1 Announce Type: new Abstract: Large language models are increasingly deployed as autonomous agents that interact with the web through browsers.
arXiv AI
Jun 4
Evaluating Reasoning Fidelity in Visual Text Generation
arXiv:2606. 04479v1 Announce Type: cross Abstract: Recent text-to-image (T2I) models can render highly legible and well-structured text within images, enabling applications including document generation and slide generation.
arXiv AI
Jul 31
See2Think: Do Multimodal Models Really Use Intermediate Visual States?
arXiv:2607. 26769v1 Announce Type: cross Abstract: Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states.