arXiv AI September 4, 2026 By Kenneth Paulsen, Florian Tambon, Mike Papadakis, Shin Yoo

Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study

The paper investigates using synthetic natural-language descriptions to contrastively pretrain small transformer encoders for code representation. By pairing generated descriptions with code in a dual-encoder setup during training and discarding them at inference, the authors achieve significant improvements over traditional pretraining baselines on most evaluated tasks. When fine‑tuned, these models match or surpass much larger zero‑shot models and remain competitive with execution‑aware supervision, indicating a scalable alternative for code embeddings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Aug 18

Invariant Pretraining for Robust Code Representations

arXiv:2608. 15412v1 Announce Type: cross Abstract: Encoder-based code representation models remain widely deployed for discriminative tasks such as clone detection and code classification, where their small size and low inference cost are decisive.

By Yifeng He, Yundi Xu, Christopher Castro Gaw Gonzalo, Zili Wang, Hao Chen

arXiv Machine Learning

2d ago

Type-IV Code Clone Detection via Layer-Wise Non-Contrastive Representation Learning

The paper introduces LWVIC4Code, a non‑contrastive, layer‑wise representation learning method for detecting Type‑IV code clones—semantically equivalent fragments that differ syntactically. It builds on the VICReg framework, adding cross‑layer consistency regularization and depth‑dependent weighting to refine semantic information across transformer layers. Experiments on Python and multi‑language datasets show that LWVIC4Code matches or outperforms contrastive baselines and zero‑shot large language models, generalizing well to Java and C# without requiring negative samples.

By Luciano Marchezan, Kevin Delcourt, Eugene Syriani, Houari Sahraoui

llms safety

arXiv Machine Learning

Aug 11

LLM-Based Embeddings for Program Analysis and Optimization

arXiv:2608. 07894v1 Announce Type: new Abstract: Recent advances have highlighted the potential of machine learning, particularly Large Language Models (LLMs), for analyzing and optimizing programs.

By Calvin Higgins, Marco Alvarez

llms rag benchmarks

arXiv Machine Learning

Jun 10

CodeAlchemy: Synthetic Code Rewriting at Scale

arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.

By Ankit Gupta, Aditya Prasad, Rameswar Panda

llms benchmarks

arXiv Machine Learning

Jun 18

CODEBLOCK: Learning to Supervise Code at the Right Granularity

arXiv:2606. 18286v1 Announce Type: new Abstract: Supervised fine-tuning of code LLMs typically applies uniform cross-entropy loss to all response tokens, implicitly assuming that every token provides equally useful learning signal.

By Zhijie Deng, Ling Li, Jinlong Pang, Kaiqin Hu, Qi Xuan, Zhaowei Zhu, Jiaheng Wei

llms fine-tuning benchmarks

arXiv AI

Aug 18

Efficient Code Embeddings from Code Generation Models

arXiv:2508. 21290v2 Announce Type: replace-cross Abstract: jina-code-embeddings is a novel code embedding model suite designed to retrieve code from natural language queries, perform technical question-answering, and identify semantically similar code snippets across programming languages.

By Daria Kryvosheieva, Saba Sturua, Michael G\"unther, Han Xiao

rag benchmarks

One email a morning, machine-written

Related stories

Invariant Pretraining for Robust Code Representations

Type-IV Code Clone Detection via Layer-Wise Non-Contrastive Representation Learning

LLM-Based Embeddings for Program Analysis and Optimization

CodeAlchemy: Synthetic Code Rewriting at Scale

CODEBLOCK: Learning to Supervise Code at the Right Granularity

Efficient Code Embeddings from Code Generation Models