arXiv AI
Jun 16
FasterPy: An LLM-based Code Execution Efficiency Optimization Framework
arXiv:2512. 22827v2 Announce Type: replace-cross Abstract: Code often suffers from performance bugs.
EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
EffiBench-X is a new multi‑language benchmark that measures the efficiency of code generated by large language models (LLMs). It covers six programming languages—Python, C++, Java, JavaScript, Ruby, and Golang—using competitive programming tasks with human‑expert solutions as efficiency baselines. Experiments show that while LLMs produce functionally correct code, they consistently fall short of human efficiency, achieving only about 62 % of human performance on average, with notable language‑specific differences.
Related stories
arXiv AI
Aug 14
CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
arXiv:2603. 14501v2 Announce Type: replace-cross Abstract: Large Language Models excel in high-resource programming languages but struggle with low-resource ones.
arXiv Computation and Language
3d ago
SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization
SWE-Opt is a new benchmark that evaluates large language models (LLMs) on repository‑level software performance optimization. It is built from 102 expert‑written optimizations from open‑source projects and measures runtime, peak memory, and Time‑Weighted Memory Usage (TWMU) under varied input data, execution conditions, and noise‑aware measurements. The study finds that current LLMs achieve few runtime gains and almost no memory improvements, whereas expert implementations deliver a 15.5× speedup and 171.3× memory reduction.
arXiv AI
Jul 21
FormulaCode: Evaluating Agentic Optimization on Large Codebases
arXiv:2603. 16011v3 Announce Type: replace-cross Abstract: Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints.
Hugging Face Trending Papers
Jun 24
Evaluating LLMs on Real-World Software Performance Optimization
Software performance optimization is a notoriously complex and manual task. Despite the growing use of Large Language Models (LLMs) for code refinement, we still lack benchmarks that capture how optimization actually happens in real-world codebases.
arXiv Computation and Language
Sep 14
Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models
arXiv:2609.12475v1 Announce Type: new Abstract: Comprehensive benchmark suites are essential for improving large language models (LLMs), but many widely used benchmarks are redundant, making evaluati...
arXiv AI
Jun 4
CodegenBench: Can LLMs Write Efficient Code Across Architectures?
arXiv:2606. 04023v1 Announce Type: cross Abstract: While large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated environments (e.
arXiv Machine Learning
Sep 1
Agnostics: Learning to Code in Any Programming Language via Reinforcement with a Universal Learning Environment
Agnostics is a language‑agnostic post‑training pipeline that uses reinforcement learning with verifiable rewards (RLVR) to improve large language models on low‑resource programming languages. By rewriting unit‑test datasets into a language‑independent I/O format, providing a short configuration for compiling and running code, and employing a single verifier that judges code by observable behavior, Agnostics eliminates the need for language‑specific engineering. Applied to Lua, Julia, R, OCaml, and Fortran, it boosts Qwen‑3 4B to rival larger models, scales to diverse families, and achieves new state‑of‑the‑art pass@1 on MultiPL‑E and a new multi‑language LiveCodeBench.
arXiv AI
Jun 9
Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs
arXiv:2606. 08840v1 Announce Type: new Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes.
arXiv Computation and Language
Oct 21
Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'
arXiv:2410.21647v5 Announce Type: replace-cross Abstract: Recently, a number of repository-level code generation benchmarks-such as CoderEval, DevEval, RepoEval, RepoBench, and LongCodeArena-have eme...
arXiv AI
Jun 19
Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
arXiv:2606. 20517v1 Announce Type: new Abstract: LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks.
arXiv AI
Jun 10
Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages
arXiv:2606. 10933v1 Announce Type: new Abstract: LLM-based coding agents are usually evaluated in familiar software settings: mainstream languages, common libraries, and public repositories.