arXiv AI June 8, 2026 By Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Liu, Jiawei Han, Benoit Schillings, Jiao Sun

SWE-IF: Aligning Code Evaluation with Human Preference

Read the original on arXiv AI →

arXiv:2510. 07315v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv AI

Aug 26

Evaluating Language Models on Cross-Language Code Functional Equivalence

The paper introduces PolyHuman, a dataset of human-written programs in C++, Java, and Python, to test whether large language models can judge functional equivalence across languages. Using this dataset, the authors evaluate several open-weight and proprietary LLMs, finding that models struggle more with harder problems, show language-specific biases, and rely partly on superficial similarity cues. They also observe run‑to‑run instability in GPT‑o4‑mini, concluding that current LLMs do not reliably capture functional equivalence within or across programming languages.

By Hui Sun, Anderson Uch\^oa, Rohit Gheyi, Wesley K. G. Assun\c{c}\~ao

llms benchmarks

arXiv AI

Jun 30

Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation

arXiv:2606. 28998v1 Announce Type: cross Abstract: Large Language Model (LLM) alignment trains an LLM using preference data to produce outputs that better meet established quality standards.

By Gias Uddin, Sanjeepan Sivapiran

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 9

Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

arXiv:2606. 08840v1 Announce Type: new Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes.

By Sayed Erfan Arefin

llms benchmarks

arXiv Machine Learning

Jun 5

Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation

arXiv:2606. 05792v1 Announce Type: cross Abstract: TLA+ has supported industrial verification at companies such as Amazon and Microsoft, yet writing correct TLA+ specifications from natural language still requires time and expertise, which limits adoption.

By Arslan Bisharat, Brian Ortiz, Eric Spencer, Khushboo Bhadauria, TaiNing Wang, George K. Thiruvathukal, Konstantin Laufer, Mohammed Abuhamad

llms safety

arXiv AI

Aug 18

ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

arXiv:2602. 15983v3 Announce Type: replace-cross Abstract: Large language models (LLMs) can translate natural language into optimization code, but silent failures pose a critical risk: code that executes and returns solver-feasible solutions may encode semantically incorrect formulations---a feasibility--correctness gap reaching 90 percentage points on compositional problems.

By Junbo Jacob Lian, Yujun Sun, Huiling Chen, Chaoyu Zhang, Hanzhang Qin, Chung-Piaw Teo

llms fine-tuning benchmarks

arXiv AI

Jun 9

Lost in the Flow with Code Talkers: Unveiling the Instruction-Tuning Tax of Large Language Models in Code Tasks

arXiv:2606. 08676v1 Announce Type: cross Abstract: AI coding assistants have significantly improved developer productivity by automatically suggesting code that aligns with user intent, and many of these tools are now integrated directly into Integrated Development Environments (IDEs).

By Shi Ying Chang, Chiok Yew Ho, Yichen Li, Yintong Huo

llms fine-tuning

One email a morning, machine-written

Related stories

Evaluating Language Models on Cross-Language Code Functional Equivalence

Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation

Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation

ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

Lost in the Flow with Code Talkers: Unveiling the Instruction-Tuning Tax of Large Language Models in Code Tasks