arXiv AI

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

BigCodeBench is a new benchmark designed to evaluate large language models (LLMs) on complex code generation tasks that require invoking multiple function calls from 139 libraries across 7 domains. Each of the 1,140 fine‑grained tasks includes an average of 5.6 test cases with 99% branch coverage, and a natural‑language variant, BigCodeBench‑Instruct, converts docstrings into concise instructions. Evaluation of 60 LLMs shows that they currently achieve at most 60% accuracy, far below the 97% human performance, highlighting a gap in handling complex instructions and compositional reasoning.

arXiv AI
Jun 6

Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation

arXiv:2512. 03086v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown remarkable capabilities in code translation, yet their performance deteriorates in low-resource programming domains such as Fortran and emerging frameworks like CUDA, where high-quality parallel data are scarce.

By Le Chen, Nuo Xu, Winson Chen, Bin Lei, Pei-Hung Lin, Dunzhi Zhou, Rajeev Thakur, Caiwen Ding, Ali Jannesari, Chunhua Liao
arXiv AI
Sep 24

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.

By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati
arXiv AI
Oct 31

Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent

The paper introduces ExLabel, an LLM‑based, self‑evolving assistant that assesses task clarity and test coverage in SWE‑bench‑style software engineering datasets. ExLabel builds an experience bank from cross‑instance knowledge, enabling context‑aware judgments that outperform static rubrics. A new benchmark, SWE‑TQ, spanning eight programming languages, demonstrates ExLabel’s superior F1 scores and its ability to filter data that improves a coding model’s success rate on SWE‑bench Verified.

By Zhonghao Jiang, Xuyang Zheng, Yanfu Yan, Zhongxin Liu, Lingfeng Bao, Shuiguang Deng
arXiv AI
1d ago

BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution

BigCodeArena is an open human evaluation platform that extends Chatbot Arena by adding an on‑the‑fly code execution environment, allowing users to run and interact with LLM‑generated code. The platform collected over 14,000 code‑centric conversation sessions across 10 large language models, 10 languages, and 8 execution environments, yielding more than 4,700 multi‑turn samples with pairwise human preferences. From this data, the authors created two benchmarks—BigCodeReward, which aligns reward models with human preferences, and AutoCodeArena, an automatic Elo rating system—to assess coding quality, finding that proprietary models such as GPT‑5, Claude‑Sonnet‑4, and Claude‑Opus‑4 still outperform newer emerging models.

By Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang, Tianyang Liu, Chen Gong, Bhupesh Bishnoi, Vaisakhi Mishra, Marek Suppa, Noah Ziems, Saiteja Utpala, Ming Xu, Guangyu Song, Kaixin Li, Yuhan Cao, Bo Liu, Zheng Liu, Sabina Abdurakhmanova, Wenhao Yu, Mengzhao Jia, Jihan Yao, Kenneth Hamilton, Kumar Shridhar, Minh Chien Vu, Dingmin Wang, Jiawei Liu, Zijian Wang, Qian Liu, Binyuan Hui, Meg Risdal, Ahsen Khaliq, Atin Sood, Zhenchang Xing, Wasi Uddin Ahmad, John Grundy, David Lo, Banghua Zhu, Xiaoning Du, Torsten Scholak, Leandro von Werra
arXiv Computation and Language
Oct 28

EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code

EffiBench-X is a new multi‑language benchmark that measures the efficiency of code generated by large language models (LLMs). It covers six programming languages—Python, C++, Java, JavaScript, Ruby, and Golang—using competitive programming tasks with human‑expert solutions as efficiency baselines. Experiments show that while LLMs produce functionally correct code, they consistently fall short of human efficiency, achieving only about 62 % of human performance on average, with notable language‑specific differences.

By Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie M. Zhang, Heming Cui, Siu-Ming Yiu, Dong Huang, See-Kiong Ng, Luu Anh Tuan
arXiv AI
Sep 28

ORCA: Evaluating LLMs on Data Science Code Translation

ORCA is a new benchmark for evaluating large language models on Data Science Code Translation (DSCT), comprising two settings: ORCA-MAIN with 1,600 grounding-level tasks across data querying, manipulation, and deep learning, and ORCA-PROJECT with 200 full-project translation tasks across seven data‑science task types. Each task includes reference translations and test cases to verify functional equivalence, and a multi‑stage quality verification process ensures task correctness. Experiments show that even state‑of‑the‑art LLMs perform poorly on DSCT, with Claude‑Opus‑4.6 achieving only 56.92% success on ORCA‑MAIN and 33.67% on ORCA‑PROJECT, while an intent‑augmented approach improves success rates by 4.80% and 5.33% respectively.

By Xiaolong Li, Jinyang Li, Bowen Qin, Ge Qu, Nan Huo, Xiaohan Xu, Shipei Lin, Reynold Cheng
arXiv AI
Oct 1

Composing Task-specific Agent Harnesses at Test Time with Reusable Primitives

The paper introduces Harness Primitives—reusable agent harness mechanisms mined from failed task trajectories—and a framework called STITCH that selects and composes these primitives into task‑specific harnesses at test time. This approach avoids generating or debugging harness code for each task, achieving up to 12‑point gains in task success over fixed harness baselines and outperforming human‑designed harnesses like Codex CLI. STITCH also demonstrates minimal test‑time overhead (2.7%) and scales efficiently with the size of the primitive library.

By Peng Kuang, Haibo Jin, Dehao Wu, Feiyang Deng, Xiaopeng Yuan, Jerry Wang, Haohan Wang