← Back to all news
arXiv AI By Tomas Balyo, Lukas Chrpa, G. Michael Youngblood

Which LLM is Best for Translating Natural Language Goals to PDDL

Read the original on arXiv AI →

The paper evaluates how well current Large Language Models can translate natural language goals, written by video game testers, into well‑formed PDDL targets for classical planning. Using a carefully designed prompt template, six state‑of‑the‑art LLMs were tested on correctness, speed, and error tendencies with real‑world benchmarks. All models achieved high correctness (>92%), with Gemini 2.5 Flash reaching 96% accuracy and the fewest false positives, while GPT‑4.1 was the fastest, yet differences in performance and occasional failures due to ambiguity and domain limits remain.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 11

Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

The paper presents an end‑to‑end pipeline for translating natural language planning descriptions into PDDL problem instances using large language models. It incorporates multiple checks—syntactic parsing, planner success, domain conformance, an LLM critic, and iterative repair—to ensure faithfulness to the original task. Experiments on Planetarium, AutoPlanBench, and curated PDDL~2.1 problems reveal that operational success can diverge from benchmark‑reference reconstruction, and that structured repair improves outcomes while PDDL~2.1 remains challenging for reference reconstruction.

By Joana Rosa, Pedro Santos, Valdemar Oliveira, Rom\~ao Silva, L. Miguel Silveira, Bruno Martins
llmsbenchmarks
More like this →
arXiv Computation and Language
Oct 23

Language Models as Higher-Order Planning Formalizers

The paper examines the limitations of using large language models (LLMs) to translate natural language planning problems into PDDL representations, noting that standard benchmarks often map directly to PDDL and may overstate success. It introduces the concept of "unraveling problems," where concise natural language descriptions correspond to large PDDL structures, and shows that traditional LLM formalizers struggle to scale on such variants. To address this, the authors propose a new paradigm, LLM-as-Higher-Order-Formalizer, where the LLM first generates a high‑level program that encapsulates recurrent logic, which then produces the full PDDL representation, thereby decoupling token output from combinatorial explosion and improving performance on complex tasks.

By Owen Jiang, Cassie Huang, Ashish Sabharwal, Li Zhang
llmsbenchmarks
More like this →
arXiv Machine Learning
Jun 5

Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation

arXiv:2606. 05792v1 Announce Type: cross Abstract: TLA+ has supported industrial verification at companies such as Amazon and Microsoft, yet writing correct TLA+ specifications from natural language still requires time and expertise, which limits adoption.

By Arslan Bisharat, Brian Ortiz, Eric Spencer, Khushboo Bhadauria, TaiNing Wang, George K. Thiruvathukal, Konstantin Laufer, Mohammed Abuhamad
llmssafety
More like this →