The paper evaluates how well open-weight large language models can repair Planning Domain Definition Language (PDDL) models using only LLMs. Experiments show that while the best LLM achieves an F1 score of 0.87—an improvement of 0.38 over a symbolic baseline—it still fails to reliably satisfy test constraints, with a mean test pass rate of only 0.82 and as low as 0.06 on the Thoughtful domain. The study concludes that current open-weight models cannot guarantee the necessary test constraint satisfaction for dependable automated model repair.
By Nader Karimi Bavandpour, Pascal Bercher
arXiv:2606. 29700v1 Announce Type: new Abstract: Planning often requires symbolic specifications that are both executable and verifiable.
By Jiamei Jiang, Jiajing Zhang, Feifei Mo, Linjing Li, Daniel Zeng
The paper evaluates how well current Large Language Models can translate natural language goals, written by video game testers, into well‑formed PDDL targets for classical planning. Using a carefully designed prompt template, six state‑of‑the‑art LLMs were tested on correctness, speed, and error tendencies with real‑world benchmarks. All models achieved high correctness (>92%), with Gemini 2.5 Flash reaching 96% accuracy and the fewest false positives, while GPT‑4.1 was the fastest, yet differences in performance and occasional failures due to ambiguity and domain limits remain.
By Tomas Balyo, Lukas Chrpa, G. Michael Youngblood
The paper examines the limitations of using large language models (LLMs) to translate natural language planning problems into PDDL representations, noting that standard benchmarks often map directly to PDDL and may overstate success. It introduces the concept of "unraveling problems," where concise natural language descriptions correspond to large PDDL structures, and shows that traditional LLM formalizers struggle to scale on such variants. To address this, the authors propose a new paradigm, LLM-as-Higher-Order-Formalizer, where the LLM first generates a high‑level program that encapsulates recurrent logic, which then produces the full PDDL representation, thereby decoupling token output from combinatorial explosion and improving performance on complex tasks.
By Owen Jiang, Cassie Huang, Ashish Sabharwal, Li Zhang
arXiv:2508.01300v2 Announce Type: replace
Abstract: AI planning provides agents the ability to act autonomously towards achieving a goal in a given environment. Recent advances in Large Language Mode...
By Ma'ayan Armony, Albert Mero\~no-Pe\~nuela, Gerard Canal
arXiv:2511. 14967v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown great promise in generating structured diagrams from natural language descriptions, particularly Mermaid sequence diagrams for software engineering.
By Basel Shbita, Farhan Ahmed, Chad DeLuca
NL2AGBench is a benchmark that evaluates how well large language models can translate English geometry problems into the formal language required by AlphaGeometry’s theorem‑proving engine. The study tests ten state‑of‑the‑art LLMs, comparing executable translation accuracy, syntactic correctness, and error types, and finds a large gap between closed‑source and open‑source models. The authors also propose an error taxonomy and test mitigation strategies such as few‑shot prompting, fine‑tuning, and human‑guided hinting, which improve performance across model families.
By Samuel Xiao, Judy Song, Rory Hu, Ziliang Zong
arXiv:2608. 16637v1 Announce Type: new Abstract: LLMs remain unreliable for long-horizon planning, often generating logically inconsistent or non-applicable plans.
By Veit Laule, Jiangtao Shuai, Manfred Hauswirth, Sonja Schimmler
arXiv:2606. 02837v1 Announce Type: cross Abstract: Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited.
By Andrea Brunello, Cristian Curaba, Luca Geatti, Michele Mignani, Angelo Montanari, Nicola Saccomanno
arXiv:2606. 30441v1 Announce Type: cross Abstract: A rigorous formalization of system requirements is a fundamental prerequisite for the verification of Multi-Agent Systems (MAS).
By Marco Aruta, Francesco Improta, Vadim Malvone, Aniello Murano, Vladana Perlic
The paper introduces DRAGO, a framework that repairs formal specifications generated by large language models (LLMs) using deterministic, formalism‑specific diagnosis instead of ungrounded LLM critiques. DRAGO selectively rewrites specifications only when the diagnosed error weight decreases, reducing regressions from 11.8% to 1.3% and improving answer accuracy by 12.9 percentage points over zero‑shot formalization and 8.7 percentage points over LLM‑diagnostic repair across five benchmarks. Additionally, the authors identified and corrected rendering defects in Planetarium that caused half of its floor‑tile split instances to be inconsistent with reference formalizations.
By Giovanni Vaccarino, Emanuele La Malfa, Dehai Min, Hanrong Zhang, Pier Luca Lanzi, Lu Cheng, Michael J. Wooldridge, Philip S. Yu, Sara Bernardini
Goldsmith is an agentic pipeline that transforms a small set of expert-annotated calibration examples into a reusable structured annotation definition treated as a trainable textual object. Candidate definitions are evaluated on the same gold examples using an executable structured loss, and a large language model editor revises high-loss failures only when loss decreases. The approach outperforms direct rewriting, OPRO, APE, and PromptBreeder in prompt-optimization tests and enhances downstream annotation across typed span, pair-level relation, and fixed-trigger event-argument tasks when combined with retrieval, score-based routing, and human review.
By Yihan Li, Hanyi Zhang, Xiaoxi Jiang, Man Guo