The paper introduces WebLoop, an execution‑grounded framework that jointly trains generation, critique, and refinement within a shared policy for functional web generation. WebLoop trains an execution‑free Critic using requirement‑level discriminability and downstream helpfulness signals, then incorporates consequence‑aware credit to improve diagnosis and repair. Experiments with Qwen3.5‑9B show significant gains on WebRise and WebGen‑Bench, with benefits that transfer to larger models and multimodal settings.
arXiv:2608. 06474v1 Announce Type: new Abstract: Large language models increasingly generate complete websites from natural-language descriptions, and reinforcement learning has become a central approach to closing their remaining functional gap.
Generating HTML from natural-language specifications is an important application of code large language models (LLMs). Despite producing renderable pages, current models often fail to ensure functiona...
The paper proposes a reinforcement learning framework that emphasizes designing feedback environments rather than solely relying on rewards. It integrates human domain knowledge via rubrics and model reasoning with hierarchical rewards to evaluate complex, partially successful outputs in open‑ended tasks. Applied to interactive web application generation, the model achieves 41.25% pass rate on MiniAppBench and 76.19% on ArtifactsBench, with ablation studies confirming the value of each component.
The paper introduces WebMRE, an offline benchmark comprising 541 tasks and 5,293 steps extracted from WebArena trajectories, designed to provide deterministic scoring for web agents without live environments. It enables the first systematic study of how guide sentences and grounded actions reinforce each other, showing that jointly decoding a guide improves element selection accuracy and that the guide acts as a causal instruction channel. The authors fine‑tune models that outperform leading zero‑shot baselines on all offline metrics.
arXiv:2608. 00335v1 Announce Type: cross Abstract: Compact web agents can reduce deployment cost, but training them poses challenges in both data collection and post-SFT reinforcement learning (RL).
The paper introduces Cross-Behavior Policy Optimization (CBPO), a method that unifies offline demonstrations and policy-generated rollouts into a single regression loss using a fixed-reference target. CBPO eliminates the need for separate supervised fine‑tuning and reinforcement learning objectives, allowing positive and negative offline completions, reference responses, and current and historical rollouts to share a common reward‑labeled format and replay buffer. Experiments on two model scales and four mathematical reasoning benchmarks show that CBPO, especially the CBPO‑Unified variant, outperforms baseline methods in avg@32 and pass@32 metrics, with historical rollouts providing additional gains.
arXiv:2608. 16156v1 Announce Type: new Abstract: Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes, making fine-grained credit assignment across multi-step interactions difficult.
WebCoder is a general frontend development agent that uses artifact‑grounded agentic reinforcement learning to learn how to compose, generate, modify, and repair code over an evolving application state. It evaluates development actions by their impact on a shared executable artifact and a verifier that provides terminal rewards, while intermediate progress redistributes rollout advantage across development segments. The system includes Webdev‑World, an executable environment that integrates code editing, execution, browser inspection, and on‑demand skills, and Webdev‑Bench, a benchmark for composed development tasks, where WebCoder‑27B outperforms several large language models on requirement satisfaction and visual/functional scores.
arXiv:2607. 27271v1 Announce Type: new Abstract: Code models are increasingly trained with execution feedback, but most training signals still stop at correctness.
Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes, making fine-grained credit assignment across multi-step interactions difficult. Existing approache...
The paper introduces Trace, a framework that transforms sparse-reward trajectories into executable walkthroughs by identifying progress anchors, propagating credit, and estimating action prerequisites. Trace compiles noisy trajectories into state‑conditioned, verifiable procedures that remove loops and detours, enabling reuse, intermediate‑state resumption, and programmatic verification. Experiments on J‑TTL, WebShop, and ScienceWorld with three open‑source LLMs show that Trace outperforms eight baselines, improving average AUC and Final‑$3$ by 30.0% and 40.5% while using fewer inference tokens.
arXiv:2608.23830v1 Announce Type: cross Abstract: RL has emerged as a powerful paradigm for enhancing the instruction following capabilities of LLMs. While existing training recipes achieve substanti...