arXiv:2512. 03086v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown remarkable capabilities in code translation, yet their performance deteriorates in low-resource programming domains such as Fortran and emerging frameworks like CUDA, where high-quality parallel data are scarce.
By Le Chen, Nuo Xu, Winson Chen, Bin Lei, Pei-Hung Lin, Dunzhi Zhou, Rajeev Thakur, Caiwen Ding, Ali Jannesari, Chunhua Liao
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.
By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati
The paper introduces ExLabel, an LLM‑based, self‑evolving assistant that assesses task clarity and test coverage in SWE‑bench‑style software engineering datasets. ExLabel builds an experience bank from cross‑instance knowledge, enabling context‑aware judgments that outperform static rubrics. A new benchmark, SWE‑TQ, spanning eight programming languages, demonstrates ExLabel’s superior F1 scores and its ability to filter data that improves a coding model’s success rate on SWE‑bench Verified.
By Zhonghao Jiang, Xuyang Zheng, Yanfu Yan, Zhongxin Liu, Lingfeng Bao, Shuiguang Deng
BigCodeArena is an open human evaluation platform that extends Chatbot Arena by adding an on‑the‑fly code execution environment, allowing users to run and interact with LLM‑generated code. The platform collected over 14,000 code‑centric conversation sessions across 10 large language models, 10 languages, and 8 execution environments, yielding more than 4,700 multi‑turn samples with pairwise human preferences. From this data, the authors created two benchmarks—BigCodeReward, which aligns reward models with human preferences, and AutoCodeArena, an automatic Elo rating system—to assess coding quality, finding that proprietary models such as GPT‑5, Claude‑Sonnet‑4, and Claude‑Opus‑4 still outperform newer emerging models.
By Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang, Tianyang Liu, Chen Gong, Bhupesh Bishnoi, Vaisakhi Mishra, Marek Suppa, Noah Ziems, Saiteja Utpala, Ming Xu, Guangyu Song, Kaixin Li, Yuhan Cao, Bo Liu, Zheng Liu, Sabina Abdurakhmanova, Wenhao Yu, Mengzhao Jia, Jihan Yao, Kenneth Hamilton, Kumar Shridhar, Minh Chien Vu, Dingmin Wang, Jiawei Liu, Zijian Wang, Qian Liu, Binyuan Hui, Meg Risdal, Ahsen Khaliq, Atin Sood, Zhenchang Xing, Wasi Uddin Ahmad, John Grundy, David Lo, Banghua Zhu, Xiaoning Du, Torsten Scholak, Leandro von Werra
arXiv:2506. 13932v3 Announce Type: replace-cross Abstract: The rise of large language models (LLMs) has led to dramatic improvements across a wide range of natural language tasks.
By Saurabh Pujar, Ira Ceka, Irene Manotas, Gail Kaiser, Baishakhi Ray, Shyam Ramji
arXiv:2601. 03808v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have achieved notable performance in code synthesis; however, data-aware augmentation remains a limiting factor, handled via heuristic design or brute-force approaches.
By Usha Shrestha, Dmitry Ignatov, Radu Timofte