← Back to all news
OpenAI Blog February 18, 2025

Introducing the SWE-Lancer benchmark

Read the original on OpenAI Blog →

Can frontier LLMs earn $1 million from real-world freelance software engineering?

Summary generated by The Flow from the publisher's feed. The full article lives at OpenAI Blog.

  • llms
  • benchmarks

Related stories

arXiv AI
Jul 17

Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utility

arXiv:2607. 14108v1 Announce Type: cross Abstract: This paper introduces tool efficiency, a new quantitative metric to evaluate the rate of useful tool calls in an LLM agent trajectory.

By Nyx Iskandar
llmsagentsbenchmarks
More like this →
arXiv AI
Jul 14

BizFinBench.v2: Towards Reliable LLMs in Finance via Real-User Data and Offline/Online Bilingual Evaluation

arXiv:2601. 06401v2 Announce Type: replace Abstract: Large language models are becoming increasingly significant in financial applications.

By Xin Guo, Rongjunchen Zhang, Guilong Lu, Xuntao Guo, Shuai Jia, Zhi Yang, Liwen Zhang
llmsbenchmarks
More like this →
arXiv AI
Jun 29

Towards Evaluation of Implicit Software World Models in Coding LLMs

arXiv:2606. 27406v1 Announce Type: cross Abstract: Software engineering, whether performed by humans or by AI agents, requires reasoning about how software behaves.

By Egor Bogomolov, Yaroslav Zharov
llmsagentsbenchmarks
More like this →
OpenAI Blog
May 6

How frontier firms are pulling ahead

OpenAI’s B2B Signals research shows how frontier enterprises deepen AI adoption, scale Codex-powered agentic workflows, and build durable competitive advantage.

agents
More like this →
arXiv AI
Jun 6

ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developer

arXiv:2606. 05548v1 Announce Type: cross Abstract: The rapid proliferation of Agent Development Kits (ADKs), SDK-level frameworks for building LLM-powered autonomous agents, has outpaced any empirical understanding of how framework choice affects agent performance.

By Jintao Huang, Xiaomin Li, Gaurav Mittal, Yu Hu
llmsagentsbenchmarks
More like this →
OpenAI Blog
Feb 23

Why we no longer evaluate SWE-bench Verified

SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress. Our analysis shows flawed tests and training leakage.

More like this →