arXiv AI

BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution

BigCodeArena is an open human evaluation platform that extends Chatbot Arena by adding an on‑the‑fly code execution environment, allowing users to run and interact with LLM‑generated code. The platform collected over 14,000 code‑centric conversation sessions across 10 large language models, 10 languages, and 8 execution environments, yielding more than 4,700 multi‑turn samples with pairwise human preferences. From this data, the authors created two benchmarks—BigCodeReward, which aligns reward models with human preferences, and AutoCodeArena, an automatic Elo rating system—to assess coding quality, finding that proprietary models such as GPT‑5, Claude‑Sonnet‑4, and Claude‑Opus‑4 still outperform newer emerging models.

arXiv AI
1d ago

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

BigCodeBench is a new benchmark designed to evaluate large language models (LLMs) on complex code generation tasks that require invoking multiple function calls from 139 libraries across 7 domains. Each of the 1,140 fine‑grained tasks includes an average of 5.6 test cases with 99% branch coverage, and a natural‑language variant, BigCodeBench‑Instruct, converts docstrings into concise instructions. Evaluation of 60 LLMs shows that they currently achieve at most 60% accuracy, far below the 97% human performance, highlighting a gap in handling complex instructions and compositional reasoning.

By Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, Leandro Von Werra
arXiv AI
Oct 31

Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent

The paper introduces ExLabel, an LLM‑based, self‑evolving assistant that assesses task clarity and test coverage in SWE‑bench‑style software engineering datasets. ExLabel builds an experience bank from cross‑instance knowledge, enabling context‑aware judgments that outperform static rubrics. A new benchmark, SWE‑TQ, spanning eight programming languages, demonstrates ExLabel’s superior F1 scores and its ability to filter data that improves a coding model’s success rate on SWE‑bench Verified.

By Zhonghao Jiang, Xuyang Zheng, Yanfu Yan, Zhongxin Liu, Lingfeng Bao, Shuiguang Deng
arXiv Machine Learning
Jun 11

FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback

arXiv:2601. 04203v2 Announce Type: replace-cross Abstract: We present FronTalk, a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generation with multi-modal feedback.

By Xueqing Wu, Zihan Xue, Da Yin, Shuyan Zhou, Kai-Wei Chang, Nanyun Peng, Yeming Wen
arXiv AI
Oct 9

SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction

SWE-Journey is a new benchmark designed to evaluate coding assistants in more realistic settings, focusing on long-horizon tasks and multi-turn interactions. It introduces a weak-to-strong synthesis pipeline to create extended coding challenges and a user-simulation agent based on four real user personas to mimic authentic code-assistance dialogues. The benchmark reveals that while models succeed on over 75% of tests with software architects, they manage less than 25% with non-coders, highlighting a significant gap in current assistant capabilities.

By Hexuan Deng, Yue Wang, Wenyu Jiang, Cheng Yang, Haolin Yang, Zhaohua Zhang, Chenchen Zhao, Beiduo Chen, Muxi Chen, Sa Zhu, Geyuan Zhu, Jianhuan Zhuo, Qiuyong Xiao, Tianwen Jiang, Jihong Zhang, Xuebo Liu
arXiv AI
Oct 16

Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

The paper introduces AXIOM, a perturbation-based framework for creating a balanced benchmark of code evaluation problems. AXIOM refines program quality by applying controlled semantic perturbations and then calibrates scores through selective manual annotation, resulting in the multilingual AXIOMBENCH dataset with 1,962 problems. This approach addresses limitations of prior benchmarks that rely on coarse binary labels or subjective human scoring.

By Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao
arXiv AI
Oct 15

Interaction-as-a-Judge

arXiv:2610.17895v2 Announce Type: cross Abstract: Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games. While humans assess...

By Wayne Chi, I-Hung Hsu, Li Chen, Wei-Lin Chiang, Ion Stoica, Anastasios Nikolas Angelopoulos, Ameet Talwalkar, Chris Donahue, Cho-Jui Hsieh