arXiv Computation and Language

BigO(Bench): Can LLMs Generate Code with Controlled Time and Space Complexity?

arXiv AI

2d ago

Complexity-Aware Evaluation of LLM Comprehension

arXiv:2609.37405v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for software engineering tasks that require understanding existing source code, including behavior...

By Ali Mohammadi Esfahani, Nafiseh Kahani, Samuel A. Ajila

llms benchmarks

arXiv AI

Jun 2

FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization

arXiv:2605. 25246v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimization problems often require a harder capability: designing scalable algorithms that exploit problem structure and outperform direct formulation-and-solve baselines.

By Minwei Kong, Chonghe Jiang, Ao Qu, Wenbin Ouyang, Zhaoming Zeng, Xiaotong Guo, Zhekai Li, Junyi Li, Yi Fan, Xinshou Zheng, Xi Jing, Yikai Zhang, Zhiwei Liang, Seonghoo Kim, Runqing Yang, Zijian Zhou, Sirui Li, Han Zheng, Wangyang Ying, Ou Zheng, Chonghuan Wang, Jinglong Zhao, Hanzhang Qin, Cathy Wu, Paul Pu Liang, Jinhua Zhao, Hai Wang

llms agents benchmarks

arXiv AI

Jun 16

FasterPy: An LLM-based Code Execution Efficiency Optimization Framework

arXiv:2512. 22827v2 Announce Type: replace-cross Abstract: Code often suffers from performance bugs.

By Yue Wu, Minghao Han, Ruiyin Li, Peng Liang, Amjed Tahir, Zengyang Li, Qiong Feng, Mojtaba Shahin

llms rag fine-tuning benchmarks

arXiv AI

Sep 15

Exploring the Potential of Diffusion Large Language Models in Code Generation

arXiv:2509.11252v3 Announce Type: replace-cross Abstract: LLMs have become the mainstream approaches to code generation. Existing LLMs mainly employ autoregressive generation, i.e. generating code to...

By Chengze Li, Yitong Zhang, Jia Li, Liyi Cai, Ge Li

llms diffusion benchmarks

arXiv AI

Jun 24

Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code

arXiv:2606. 23690v1 Announce Type: cross Abstract: Autoregressive (AR) language models have driven significant progress in automated software engineering, enabling powerful code generation and assistance systems.

By Kishan Maharaj, Ashita Saxena, Srikanth Tamilselvam

llms agents diffusion

arXiv AI

Jun 4

OckBench: Measuring the Efficiency of LLM Reasoning

arXiv:2511. 05722v3 Announce Type: replace-cross Abstract: Large language models (LLMs) such as GPT-5 and Gemini 3 have pushed the frontier of automated reasoning and code generation.

By Zheng Du, Hao Kang, Song Han, Tushar Krishna, Ligeng Zhu

llms benchmarks

arXiv AI

Sep 24

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.

By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati

llms benchmarks

arXiv AI

Jul 22

FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis

arXiv:2607. 18260v1 Announce Type: new Abstract: We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis.

By Soham Dan

llms rag agents benchmarks

arXiv AI

Jun 9

Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

arXiv:2606. 08840v1 Announce Type: new Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes.

By Sayed Erfan Arefin

llms benchmarks

arXiv AI

Aug 14

CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language

arXiv:2603. 14501v2 Announce Type: replace-cross Abstract: Large Language Models excel in high-resource programming languages but struggle with low-resource ones.

By Junhang Cheng, Fang Liu, Jia Li, Chengru Wu, Nanxiang Jiang, Li Zhang

llms rag agents benchmarks

arXiv AI

Jul 3

Rethinking Complexity Metrics for LLM-Integrated Applications: Beyond Source Code

arXiv:2607. 01903v1 Announce Type: new Abstract: LLM-integrated applications blend natural language prompts with program code, and much of their runtime behavior originates in the prompt layer rather than in the code itself.

By Zihao Xu, Yuekang Li, Gelei Deng, Yi Liu, Zhenchang Xing

llms

arXiv Machine Learning

Jul 10

Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation

arXiv:2607. 07748v1 Announce Type: new Abstract: Large Language Models achieve strong code generation for high resource languages like Python and Java but suffer sharp performance drops on Low-Resource Programming Languages~(LRPLs) such as Julia.

By Didula Samaraweera, Anjana Supun, Srinath Perera

llms reinforcement-learning fine-tuning benchmarks

Related stories