Training CodeParrot 🦜 from Scratch

Personal Copilot: Train Your Own Coding Assistant

Text and code embeddings by contrastive pre-training

StarCoder: A State-of-the-Art LLM for Code

Creating a Coding Assistant with StarCoder

Performance, Efficiency and Collapse -- Advantages and Challenges in Offline Post-training of Code LLMs

The paper investigates whether reinforcement‑learning post‑training of code‑generating large language models can be done entirely offline using existing datasets, avoiding costly online code generation and GPU‑CPU communication. Experiments show that a few hours of offline RL can substantially boost zero‑shot code generation performance across models from 0.5 B to 7 B parameters, though the magnitude of improvement differs by model family.

Using Local Coding Agents

Using Open-Weight Models in Local Coding Harnesses as an Alternative to Claude Code and Codex Subscriptions

Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study

The paper investigates using synthetic natural-language descriptions to contrastively pretrain small transformer encoders for code representation. By pairing generated descriptions with code in a dual-encoder setup during training and discarding them at inference, the authors achieve significant improvements over traditional pretraining baselines on most evaluated tasks. When fine‑tuned, these models match or surpass much larger zero‑shot models and remain competitive with execution‑aware supervision, indicating a scalable alternative for code embeddings.

Invariant Pretraining for Robust Code Representations

arXiv:2608. 15412v1 Announce Type: cross Abstract: Encoder-based code representation models remain widely deployed for discriminative tasks such as clone detection and code classification, where their small size and low inference cost are decisive.

CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

arXiv:2609.22068v1 Announce Type: new Abstract: Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich sourc...

Evaluating large language models trained on code

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies.

The Age of Machine Learning As Code Has Arrived