DecompGym is a benchmark that tests large language model agents on designing Benders decomposition and column generation algorithms. The benchmark contains 81 instances across nine problem families, requiring agents to produce solvers that run within 60 seconds, pass a method audit by an LLM judge, and achieve an objective within 1% of the optimal value. Evaluation of 12 GPT, Claude, and Gemini models shows that while some models solve all instances on the first try, many fail due to missing or suboptimal solutions, and reflection-based debugging improves success rates compared to simple regeneration.
arXiv:2607. 20499v1 Announce Type: new Abstract: Large Language Models generate plausible backend code, but a single-pass paradigm provides no guarantee of correctness or runtime reliability.
arXiv:2608. 14863v1 Announce Type: cross Abstract: LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified.
arXiv:2607. 18161v1 Announce Type: cross Abstract: Coding agents are increasingly used to accelerate code generation in many downstream tasks, such as fixing bugs, building applications, and prototyping.
The paper introduces OptTips, a knowledge base of 50 expert optimization modeling techniques, and OptDachshund, a multi‑agent framework that generates mathematical models and solver code from natural‑language problem descriptions. Using these tools, the authors create the EfficientOpt benchmark, comprising 561 expert‑reviewed tasks with paired reference implementations, to evaluate large language models (LLMs) on both correctness and computational efficiency. Their experiments with 11 LLMs show a consistent efficiency gap: even when LLMs produce correct solutions, the resulting programs often take longer to solve than expert‑crafted counterparts, highlighting the need to assess both accuracy and runtime performance in LLM‑based optimization modeling.
Humanize is a multi‑agent workflow for agentic coding that uses explicit, mechanically enforced decisions—called judgement engineering—to separate planning, implementation, review, and learning. A human approves a plan contract, a builder agent implements it in rounds, and a reviewer agent from another vendor decides completion, with deterministic hooks and 72 mechanical gates ensuring that a defect survives only if both models miss it. The system has been deployed in over 68 versions in 108 days, earned 1,468 GitHub stars, and powered applications such as a 567‑file gem5 build‑system migration, Kernel Design Agents that excelled in MLSys 2026 contests, and Humanize Olympiad Agents that achieved top scores in multiple international competitions and high rankings on PutnamBench and Lean‑Eval. whyItMatters":"Humanize demonstrates that a structured, multi‑agent approach with explicit mechanical gates can improve reliability and performance in automated code generation, as evidenced by its adoption, star count, and success in competitive benchmarks."
The paper introduces ARTEMIS, a no-code evolutionary optimization platform that automatically tunes large language model (LLM) agents by jointly optimizing prompts, tool descriptions, and parameters using semantically-aware genetic operators. Starting from a benchmark script and natural language goals, ARTEMIS discovers configurable components, extracts performance signals from execution logs, and evolves configurations without architectural changes. Experiments on four agent systems show significant gains: a 13.6% increase in acceptance rate for the ALE Agent, a 10.1% performance boost for the Mini‑SWE Agent, a 36.9% token‑reduction for the CrewAI Agent, and a 22% accuracy improvement for the MathTales‑Teacher Agent using a smaller open‑source model.
The paper introduces RuVerBench, a benchmark with 2,458 instances for evaluating the reliability of Large Language Models acting as judges (LaaJ) in verifying rubric compliance within agentic scenarios such as deep research and agentic coding. It systematically meta‑evaluates frontier LLMs, revealing that even the most advanced models perform well yet still produce substantial noise. The study also examines how prompt design, batching, and majority voting affect verification accuracy, noting that weaker models are more prompt‑sensitive, batched verification trades accuracy for efficiency, and majority voting offers diminishing returns.
arXiv:2606. 19787v1 Announce Type: new Abstract: Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear.
AgentSWE is a new benchmark for agent software engineering that evaluates coding agents across three lifecycle stages: creating an agent from a natural‑language requirement, editing existing agent codebases, and optimizing toward a measurable target. The benchmark consists of 25 commission‑style tasks, where builders iterate on a few illustrative cases and are then tested on held‑out cases that cover the full requirement, with hard pass/fail gates or objective‑based grading. Experiments show that current coding agents fail to meet the requirements, revealing common failure modes such as overfitting, incomplete runtimes, and unsupported completion claims.
arXiv:2605. 25246v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimization problems often require a harder capability: designing scalable algorithms that exploit problem structure and outperform direct formulation-and-solve baselines.
arXiv:2407. 19633v4 Announce Type: replace Abstract: Optimization problems are pervasive in sectors from manufacturing and distribution to healthcare.
The paper investigates how quantization affects coding agents that generate and revise programs. It compares BF16, 4‑bit AWQ, and GPTQ quantization methods on Qwen2.5‑Coder‑Instruct and Devstral‑Small‑2507 models across various sizes and seeds, finding no significant loss in final accuracy. However, quantization changes behavior: it alters one‑shot versus loop‑based solve rates, tool‑use patterns, and failure modes such as compilation errors and runtime issues.