arXiv AI

Which LLM is Best for Translating Natural Language Goals to PDDL

The paper evaluates how well current Large Language Models can translate natural language goals, written by video game testers, into well‑formed PDDL targets for classical planning. Using a carefully designed prompt template, six state‑of‑the‑art LLMs were tested on correctness, speed, and error tendencies with real‑world benchmarks. All models achieved high correctness (>92%), with Gemini 2.5 Flash reaching 96% accuracy and the fewest false positives, while GPT‑4.1 was the fastest, yet differences in performance and occasional failures due to ambiguity and domain limits remain.

arXiv AI
Sep 11

Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

The paper presents an end‑to‑end pipeline for translating natural language planning descriptions into PDDL problem instances using large language models. It incorporates multiple checks—syntactic parsing, planner success, domain conformance, an LLM critic, and iterative repair—to ensure faithfulness to the original task. Experiments on Planetarium, AutoPlanBench, and curated PDDL~2.1 problems reveal that operational success can diverge from benchmark‑reference reconstruction, and that structured repair improves outcomes while PDDL~2.1 remains challenging for reference reconstruction.

By Joana Rosa, Pedro Santos, Valdemar Oliveira, Rom\~ao Silva, L. Miguel Silveira, Bruno Martins
arXiv Machine Learning
Jun 5

Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation

arXiv:2606. 05792v1 Announce Type: cross Abstract: TLA+ has supported industrial verification at companies such as Amazon and Microsoft, yet writing correct TLA+ specifications from natural language still requires time and expertise, which limits adoption.

By Arslan Bisharat, Brian Ortiz, Eric Spencer, Khushboo Bhadauria, TaiNing Wang, George K. Thiruvathukal, Konstantin Laufer, Mohammed Abuhamad
arXiv AI
Jun 6

Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation

arXiv:2512. 03086v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown remarkable capabilities in code translation, yet their performance deteriorates in low-resource programming domains such as Fortran and emerging frameworks like CUDA, where high-quality parallel data are scarce.

By Le Chen, Nuo Xu, Winson Chen, Bin Lei, Pei-Hung Lin, Dunzhi Zhou, Rajeev Thakur, Caiwen Ding, Ali Jannesari, Chunhua Liao
arXiv AI
Aug 19

LLM-Only PDDL Domain Repair with Open-Weight Models

The paper evaluates how well open-weight large language models can repair Planning Domain Definition Language (PDDL) models using only LLMs. Experiments show that while the best LLM achieves an F1 score of 0.87—an improvement of 0.38 over a symbolic baseline—it still fails to reliably satisfy test constraints, with a mean test pass rate of only 0.82 and as low as 0.06 on the Thoughtful domain. The study concludes that current open-weight models cannot guarantee the necessary test constraint satisfaction for dependable automated model repair.

By Nader Karimi Bavandpour, Pascal Bercher
arXiv AI
Aug 28

STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration

The paper introduces STEP, a State‑Aware Task Estimator and Planner that uses multi‑modal large language models to explicitly estimate system states and predict state transitions during task planning. By forecasting future states alongside actions, STEP reduces hallucinated actions and improves task‑convergent planning. In a simulated robot assembly task, STEP outperforms the state‑of‑the‑art by 32.8% in action executability and 14.8% in final‑state error.

By Maitrey Gramopadhye, Prakash Baskaran, Xiao Liu, Songpo Li, Soshi Iba
arXiv Computer Vision
1d ago

GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

arXiv:2609.25001v1 Announce Type: new Abstract: Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, a...

By Yiran Wang, Xingyilang Yin, Junfu Pu, Guangzhi Wang, Kaifeng Li, Mingyu Ouyang, Huiqiang Sun, Lingen Li, Cheng Cheng, Wangbo Yu, Honghao Chen, Xiaodong Cun, Chi-Man Pun, Zhiguo Cao, Ying Shan