WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks

WebArxiv is a reproducible benchmark designed to evaluate multimodal web agents on arXiv-related tasks. It consists of 510 static, time‑invariant tasks that require multi‑constraint paper retrieval, fine‑grained content extraction, and cross‑paper comparison, each with a deterministic ground truth. The benchmark highlights challenges for foundation‑model agents, such as over‑reliance on fixed interaction histories, and introduces a lightweight dynamic‑memory mechanism to improve adaptive retrieval and reasoning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• agents
• multimodal
• benchmarks
• safety