arXiv AI By Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, Leandro Von Werra

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

Read the original on arXiv AI →

BigCodeBench is a new benchmark designed to evaluate large language models (LLMs) on complex code generation tasks that require invoking multiple function calls from 139 libraries across 7 domains. Each of the 1,140 fine‑grained tasks includes an average of 5.6 test cases with 99% branch coverage, and a natural‑language variant, BigCodeBench‑Instruct, converts docstrings into concise instructions. Evaluation of 60 LLMs shows that they currently achieve at most 60% accuracy, far below the 97% human performance, highlighting a gap in handling complex instructions and compositional reasoning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Jun 6

Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation

arXiv:2512. 03086v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown remarkable capabilities in code translation, yet their performance deteriorates in low-resource programming domains such as Fortran and emerging frameworks like CUDA, where high-quality parallel data are scarce.

By Le Chen, Nuo Xu, Winson Chen, Bin Lei, Pei-Hung Lin, Dunzhi Zhou, Rajeev Thakur, Caiwen Ding, Ali Jannesari, Chunhua Liao
arXiv AI
Sep 24

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.

By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati
arXiv AI
Oct 31

Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent

The paper introduces ExLabel, an LLM‑based, self‑evolving assistant that assesses task clarity and test coverage in SWE‑bench‑style software engineering datasets. ExLabel builds an experience bank from cross‑instance knowledge, enabling context‑aware judgments that outperform static rubrics. A new benchmark, SWE‑TQ, spanning eight programming languages, demonstrates ExLabel’s superior F1 scores and its ability to filter data that improves a coding model’s success rate on SWE‑bench Verified.

By Zhonghao Jiang, Xuyang Zheng, Yanfu Yan, Zhongxin Liu, Lingfeng Bao, Shuiguang Deng
arXiv AI
1d ago

BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution

BigCodeArena is an open human evaluation platform that extends Chatbot Arena by adding an on‑the‑fly code execution environment, allowing users to run and interact with LLM‑generated code. The platform collected over 14,000 code‑centric conversation sessions across 10 large language models, 10 languages, and 8 execution environments, yielding more than 4,700 multi‑turn samples with pairwise human preferences. From this data, the authors created two benchmarks—BigCodeReward, which aligns reward models with human preferences, and AutoCodeArena, an automatic Elo rating system—to assess coding quality, finding that proprietary models such as GPT‑5, Claude‑Sonnet‑4, and Claude‑Opus‑4 still outperform newer emerging models.

By Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang, Tianyang Liu, Chen Gong, Bhupesh Bishnoi, Vaisakhi Mishra, Marek Suppa, Noah Ziems, Saiteja Utpala, Ming Xu, Guangyu Song, Kaixin Li, Yuhan Cao, Bo Liu, Zheng Liu, Sabina Abdurakhmanova, Wenhao Yu, Mengzhao Jia, Jihan Yao, Kenneth Hamilton, Kumar Shridhar, Minh Chien Vu, Dingmin Wang, Jiawei Liu, Zijian Wang, Qian Liu, Binyuan Hui, Meg Risdal, Ahsen Khaliq, Atin Sood, Zhenchang Xing, Wasi Uddin Ahmad, John Grundy, David Lo, Banghua Zhu, Xiaoning Du, Torsten Scholak, Leandro von Werra