← Back to all news
arXiv AI By Giovanni Vaccarino, Emanuele La Malfa, Dehai Min, Hanrong Zhang, Pier Luca Lanzi, Lu Cheng, Michael J. Wooldridge, Philip S. Yu, Sara Bernardini

Repairing LLM-Generated Specifications with Deterministic Diagnosis

Read the original on arXiv AI →

The paper introduces DRAGO, a framework that repairs formal specifications generated by large language models (LLMs) using deterministic, formalism‑specific diagnosis instead of ungrounded LLM critiques. DRAGO selectively rewrites specifications only when the diagnosed error weight decreases, reducing regressions from 11.8% to 1.3% and improving answer accuracy by 12.9 percentage points over zero‑shot formalization and 8.7 percentage points over LLM‑diagnostic repair across five benchmarks. Additionally, the authors identified and corrected rendering defects in Planetarium that caused half of its floor‑tile split instances to be inconsistent with reference formalizations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Aug 18

ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

arXiv:2602. 15983v3 Announce Type: replace-cross Abstract: Large language models (LLMs) can translate natural language into optimization code, but silent failures pose a critical risk: code that executes and returns solver-feasible solutions may encode semantically incorrect formulations---a feasibility--correctness gap reaching 90 percentage points on compositional problems.

By Junbo Jacob Lian, Yujun Sun, Huiling Chen, Chaoyu Zhang, Hanzhang Qin, Chung-Piaw Teo
llmsfine-tuningbenchmarks
More like this →
arXiv AI
Sep 2

Predicting Program Exit Code with LLMs and Programming Language Semantics

The paper introduces Program Executability Prediction (PrEx), a task that asks large language models (LLMs) to determine whether a program is semantically valid or invalid and, if invalid, to identify the violated formal rule. To evaluate this, the authors create a dataset of systematically generated invalid programs derived from valid ones and test open‑source coding LLMs across different semantic formalisms, semantic shifts, and program splits (human‑written, LLM‑translated, fuzzer‑generated). Results show that LLMs rely more on pre‑training priors than on the provided semantics, performing poorly on modified semantics and with increasing program complexity.

By Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric
llms
More like this →