Evaluating large language models trained on code

Efficient training of language models to fill in the middle

Learn2Zinc: Fine-tuning Small Language Models for Text-to-Model Translation in MiniZinc

arXiv:2607. 20456v1 Announce Type: cross Abstract: Large language models excel at code generation for mainstream programming languages but struggle with rare, domain-specific languages such as MiniZinc, a constraint modeling language for combinatorial problems.

Uncovering Competency Gaps in Large Language Models and Their Benchmarks

arXiv:2512. 20638v2 Announce Type: replace-cross Abstract: The evaluation of large language models relies heavily on standardized benchmarks.

FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis

arXiv:2607. 18260v1 Announce Type: new Abstract: We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis.

Matryoshka Language Model Suites

arXiv:2608. 09703v1 Announce Type: new Abstract: Training a language model suite classically requires training each model separately and serving them independently.

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

arXiv:2507. 11059v3 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset.