arXiv AI

Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking

arXiv:2606. 15673v1 Announce Type: new Abstract: Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and offering little guidance on improvement.

arXiv AI

Aug 11

CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception

arXiv:2608. 08392v1 Announce Type: new Abstract: Large language models are increasingly deployed as autonomous agents that interact with the web through browsers.

By Zejun Xu, Taiyi Chen, Jin Li, Yongtong Gu, Qi Cheng, Aixuan Lv, Shuai Zhu, Pengfei Zhu, Kaichen Yang, Boyu Sun, Yixian Yang, Mulong Xie, Xin Liu, Dagang Li, Xiaoteng Ma, Hongru Wang

llms agents benchmarks

arXiv AI

Jun 18

WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks

arXiv:2604. 06367v2 Announce Type: replace-cross Abstract: Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries.

By Guruprasad Viswanathan Ramesh, Asmit Nayak, Basieem Siddique, Kassem Fawaz

llms agents multimodal benchmarks safety

arXiv AI

Aug 11

WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks

arXiv:2506. 01952v2 Announce Type: replace-cross Abstract: Powered by large language models (LLMs), web browsing agents operate graphical user interfaces in a human-like manner, offering a transparent and general framework for automating web-based tasks.

By Atsuyuki Miyai, Zaiying Zhao, Kazuki Egashira, Atsuki Sato, Tatsumi Sunada, Shota Onohara, Hiromasa Yamanishi, Mashiro Toyooka, Kunato Nishina, Ryoma Maeda, Kiyoharu Aizawa, Toshihiko Yamasaki

llms agents benchmarks

arXiv AI

Jun 2

AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.

By Shengda Fan, Xuyan Ye, Yupeng Huo, Zhi-Yuan Chen, Yiju Guo, Shenzhi Yang, Wenkai Yang, Shuqi Ye, Jingwen Chen, Haotian Chen, Xin Cong, Yankai Lin

llms agents benchmarks

arXiv AI

Jul 14

AgentAbstain: Do LLM Agents Know When Not to Act?

arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.

By Xun Liu, Yi Evie Zhang, Vira Kasprova, Parisa Rabbani, Pardis Sadat Zahraei, Tianyu Zhang, Ali Ebrahimpour-Boroojeny, Varun Chandrasekaran

llms agents benchmarks

arXiv AI

Jun 17

Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory

arXiv:2510. 19838v2 Announce Type: replace Abstract: Autonomous web agents powered by large language models (LLMs) show strong potential for performing goal-oriented tasks such as information retrieval, report generation, and online transactions.

By Shiqi He, Yue Cui, Xinyu Ma, Yaliang Li, Bolin Ding, Mosharaf Chowdhury

llms agents robotics benchmarks

arXiv Machine Learning

Jun 16

Agent trajectories as programs: fingerprinting and programming coding-agent behavior

arXiv:2606. 16988v1 Announce Type: cross Abstract: Benchmark scores tell you what an agent got right; they do not tell you how it got there.

By Hamidah Oderinwale

agents benchmarks

arXiv AI

Jun 17

Beyond Domains: Reusing Web Skills via Transferable Interaction Patterns

arXiv:2606. 17645v1 Announce Type: new Abstract: Large language model (LLM) web agents are usually deployed as tool callers: each turn, the model reads a fresh page observation and emits one structured tool action.

By Shiqi He, Yue Cui, Feijie Wu, Xinyu Ma, Jiaheng Lu, Yaliang Li, Bolin Ding, Mosharaf Chowdhury

llms agents benchmarks

arXiv AI

Jul 9

AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

arXiv:2607. 06624v1 Announce Type: new Abstract: We present AgentLens, a production-assessed benchmark for interactive code agents.

By Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, Sergey Nikolenko

llms agents benchmarks

arXiv AI

Jun 15

SANA: What Matters for QA Agents over Massive Data Lakes?

arXiv:2606. 13904v1 Announce Type: cross Abstract: Exploratory question answering (EQA) over data lakes requires an LLM agent to discover relevant sources, analyze retrieved data, and adapt its actions based on intermediate results.

By Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang, Eugene Wu

llms agents nlp benchmarks

arXiv AI

Aug 10

WebGrader: Training LLMs for Web Development with Self-Evolving Programmatic Grader

arXiv:2608. 06474v1 Announce Type: new Abstract: Large language models increasingly generate complete websites from natural-language descriptions, and reinforcement learning has become a central approach to closing their remaining functional gap.

By Boshui Chen, Huiping Liu, Shaolei Zhang

llms agents reinforcement-learning multimodal

arXiv AI

Aug 11

$A^2E$ : An End-to-End Agent Auditing Engine

arXiv:2608. 07346v2 Announce Type: replace Abstract: With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains.

By Haoning Wang, Mingxun Zhang, Chenyue Yu, Yingjun Shang, Xia Hu, Guanchu Wang, Na Zou

llms agents

Related stories