arXiv Computation and Language

VibeJam: A User Study Platform for Web Development with Agents

arXiv Machine Learning
Jun 11

FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback

arXiv:2601. 04203v2 Announce Type: replace-cross Abstract: We present FronTalk, a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generation with multi-modal feedback.

By Xueqing Wu, Zihan Xue, Da Yin, Shuyan Zhou, Kai-Wei Chang, Nanyun Peng, Yeming Wen
arXiv AI
Jun 10

How can we assess human-agent interactions? Case studies in software agent design

arXiv:2510. 09801v3 Announce Type: replace Abstract: While benchmarks measure the accuracy of LLM-powered agents, they mostly assume full automation, failing to represent the collaborative nature of real-world use cases.

By Valerie Chen, Rohit Malhotra, Xingyao Wang, Juan Michelini, Xuhui Zhou, Aditya Bharat Soni, Hoang H. Tran, Calvin Smith, Ameet Talwalkar, Graham Neubig
arXiv AI
Jul 21

ClawBench: Can AI Agents Complete Everyday Online Tasks?

arXiv:2604. 08523v2 Announce Type: replace-cross Abstract: AI agents may be able to assist with emails and documents, but can they reliably complete everyday online workflows on real websites?

By Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Zhuofeng Li, Xingwei Qu, Zhengkang Guo, Yuanzhe Shen, Dingjie Song, Han Zhou, Tuney Zheng, Xian Wu, Hao Yu, Songcheng Cai, Yi Lu, Yunzhuo Hao, Minyi Lei, Liang Chen, Kai Zou, Huifeng Yin, Wendong Xu, Dongfu Jiang, Ping Nie, Jiaheng Liu, Wenhu Chen, Kelsey R. Allen
arXiv AI
Aug 20

Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025

The paper titled "Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025" examines how experienced developers employ AI agents in software development. Through field observations and surveys, it finds that developers value agents for productivity but maintain control over design and implementation to ensure quality. They use agents as collaborative tools rather than full delegation, selecting tasks based on suitability and leveraging their expertise to guide agent behavior.

By Ruanqianqian Huang, Avery Reyna, Sorin Lerner, Haijun Xia, Brian Hempel
arXiv AI
Aug 11

WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks

arXiv:2506. 01952v2 Announce Type: replace-cross Abstract: Powered by large language models (LLMs), web browsing agents operate graphical user interfaces in a human-like manner, offering a transparent and general framework for automating web-based tasks.

By Atsuyuki Miyai, Zaiying Zhao, Kazuki Egashira, Atsuki Sato, Tatsumi Sunada, Shota Onohara, Hiromasa Yamanishi, Mashiro Toyooka, Kunato Nishina, Ryoma Maeda, Kiyoharu Aizawa, Toshihiko Yamasaki
arXiv AI
Jun 8

SW-$A^2$-Bench: Benchmarking Autonomous Software Agent Generation for Agentic Web

arXiv:2604. 04226v2 Announce Type: replace-cross Abstract: The Agentic Web is emerging as a paradigm in which autonomous software agents interact with online resources and with each other to accomplish user goals.

By Linyao Chen, Bo Huang, Qinlao Zhao, Shuai Shao, Zhi Han, Zicai Cui, Ziheng Zhang, Guangtao Zeng, Wenzheng Tang, Yikun Wang, Yuanjian Zhou, Zimian Peng, Yong Yu, Weiwen Liu, Hiroki Kobayashi, Weinan Zhang