arXiv AI

Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Tasks

The study investigates how users interact with ChatGPT for code generation beyond simple function-level tasks, focusing on project-level benchmarks that involve multi-class dependencies. A user study with 36 participants examined prompting patterns, screen recordings, and chat logs to identify Human‑LLM Interaction (HLI) features that influence productivity. The results highlight three consistently supportive HLI features, five guidelines to boost productivity, and a taxonomy of 29 runtime and logic errors with mitigation strategies.

arXiv AI
3d ago

The Impact of Large Language Models (LLMs) on Code Review Process

The study examines how GPT-assisted pull requests (PRs) influence the GitHub code review workflow. Using a dataset of 25,473 PRs, the authors identified GPT involvement with a 95% accurate heuristic and compared resolution times and phase-specific performance between GPT-assisted and non-assisted PRs. Results show that GPT reduces median resolution time by over 60%, cuts review time by 33%, and shortens pre-acceptance waiting by 87%, with developers mainly using GPT for code optimization, bug fixing, and documentation.

By Antonio Collante, Samuel Abedu, SayedHassan Khatoonabadi, Ahmad Abdellatif, Ebube Alor, Emad Shihab
arXiv AI
Jun 9

Lost in the Flow with Code Talkers: Unveiling the Instruction-Tuning Tax of Large Language Models in Code Tasks

arXiv:2606. 08676v1 Announce Type: cross Abstract: AI coding assistants have significantly improved developer productivity by automatically suggesting code that aligns with user intent, and many of these tools are now integrated directly into Integrated Development Environments (IDEs).

By Shi Ying Chang, Chiok Yew Ho, Yichen Li, Yintong Huo
arXiv AI
Aug 11

How to Ask the AI: A User Perspective Survey for Large Language Model Prompting

arXiv:2608. 07494v1 Announce Type: cross Abstract: AI tools like ChatGPT and DeepSeek, powered by Large Language Models (LLMs), allow users to obtain instant and effective content responses simply by typing requests, such as ``plan a three-day Vienna trip'', ``solve the attached mathematical problem'', ``draft an email to inquire review progress'', etc.

By Yiqun Zhang, Yunfan Zhang, Mingjie Zhao, Sen Feng, Yiu-ming Cheung
arXiv AI
Oct 12

Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development

The paper introduces ReqClarifyBench, an interactive benchmark for evaluating how well large language models (LLMs) can clarify vague user intent into clear requirements in repository-level software development. It includes 341 tasks across game, miniapp, and web development, each with a vague intent, a human-validated requirement document, and code evaluation artifacts, and uses LLM-based user simulators with diverse personas. The study evaluates six LLMs on clarification effectiveness and requirement quality, finding that stronger clarification does not always yield better documents, performance varies by persona, and coding-agent harnesses do not improve document quality over base LLMs.

By Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, Pinjia He