arXiv AI
Jul 8
UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
arXiv:2607. 06306v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated growing competence in web page generation.
UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence.
Related stories
arXiv AI
Jun 12
VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents
arXiv:2605. 26144v2 Announce Type: replace-cross Abstract: We present VISTA (VIsual Spec-To-App Benchmark), a benchmark for evaluating the end-to-end web-app generation capabilities of LLM-based agents.
arXiv Computer Vision
Sep 3
Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development
RILA is an execution‑driven agent that integrates browser rendering into the generation loop for interactive web development. It uses an Action Interaction Verification module to replay reference interactions on generated pages, collecting execution‑aware observations, and an Execution‑aware Rendering Score to jointly assess interaction correctness and visual fidelity during iterative optimization. A data synthesis pipeline further augments training data, enabling RILA to significantly improve interaction and visual quality across foundation models, even outperforming larger one‑shot generators.
arXiv AI
Jun 2
Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages
arXiv:2606. 00154v1 Announce Type: cross Abstract: Recent advancements in multimodal large language models (MLLMs) have achieved remarkable progress in multimodal reasoning and code generation, catalyzing a new paradigm for front-end development.
arXiv AI
Aug 11
Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents
arXiv:2508. 04412v3 Announce Type: replace Abstract: The advent of large language models (LLMs) has sparked an evolution of autonomous web browsing agents: given a web browsing task and serialised user interface (UI) state, an LLM is expected to suggest input actions that incrementally solve the given task.
arXiv Computation and Language
Sep 1
Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence
The paper surveys Multimodal Code Intelligence, focusing on tasks where code is generated, edited, refined, or reasoned about under visually grounded inputs such as screenshots, charts, and videos. It categorizes the field by the role of code—rendered artifact, editable structure, intermediate reasoning trace, or executable tool interface—and organizes benchmarks into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks. The authors argue that reliable evaluation must include evidence of semantics and interaction beyond visual fidelity, and propose four verification-centered research directions to advance the field toward evidence-grounded executable systems.
arXiv AI
Jul 14
MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation
arXiv:2607. 10079v1 Announce Type: new Abstract: Digital Adoption Platforms (DAPs) are embedded overlays widely used on web systems to guide users through operations inside a page, helping them get started with unfamiliar interfaces quickly.
arXiv AI
Aug 3
Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation
arXiv:2607. 28645v1 Announce Type: cross Abstract: Recent multimodal large language models can convert visual designs directly into executable code, but real mobile products require multiple screenshots to become a buildable codebase with shared components and working navigation.
arXiv AI
Aug 7
AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents
arXiv:2608. 05891v1 Announce Type: new Abstract: Mobile GUI agents can operate apps through pixel perception and touch actions, making them a promising interface for collecting and improving long-horizon mobile interaction policies.
arXiv AI
Jun 17
LongWebBench: Evaluating Structural and Functional Webpage Generation in Long-Horizon Settings
arXiv:2606. 17727v1 Announce Type: new Abstract: Recent vision-language models (VLMs) have shown promising progress in generating webpages from visual inputs, yet existing evaluations mainly focus on short, single-screen, and largely static webpages.
arXiv AI
Jun 12
Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach
arXiv:2606. 13192v1 Announce Type: new Abstract: User experience (UX) centered on usability, perceived consistency, and functional clarity is fundamental to real-world user interfaces (UI).
arXiv Computer Vision
2d ago
AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
arXiv:2610.01215v1 Announce Type: new Abstract: GUI agents require high-quality interaction trajectories to learn how software environments respond to actions, maintain state, and support multi-step...