arXiv AI
Sep 2

Predicting Program Exit Code with LLMs and Programming Language Semantics

The paper introduces Program Executability Prediction (PrEx), a task that asks large language models (LLMs) to determine whether a program is semantically valid or invalid and, if invalid, to identify the violated formal rule. To evaluate this, the authors create a dataset of systematically generated invalid programs derived from valid ones and test open‑source coding LLMs across different semantic formalisms, semantic shifts, and program splits (human‑written, LLM‑translated, fuzzer‑generated). Results show that LLMs rely more on pre‑training priors than on the provided semantics, performing poorly on modified semantics and with increasing program complexity.

By Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric
arXiv AI
Oct 31

Post-Training Large Language Models with LLMON

Large Language Models (LLMs) treat all context tokens uniformly, which creates prompt injection vulnerabilities and unreliable instruction execution. The paper introduces LLMON (LLM Object Notation), a post‑training framework that separates executable instructions from passive data and requires explicit identifier‑based execution. Using a 4.6 B‑token structured corpus and post‑training on Granite‑4.1 models, LLMON reduces indirect prompt‑injection success rates from 53–100 % to 0.02–5.6 % while maintaining general capabilities, and provides attention‑based interpretability metrics to diagnose failure modes.

By Basel Shbita, Michael Hind, Chad DeLuca, Nathan Fulton, Shubhi Asthana, Farhan Ahmed, Anisa Halimi, Yada Zhu, Mauro Martino, David Cox, Dan Gutfreund