arXiv AI By Md Rakib Hossain Misu, Iris Ma, Cristina V. Lopes

Spec-Harness: Measuring and Improving Behavioral Adequacy of LLM-Synthesized Formal Specifications

Read the original on arXiv AI →

Spec‑Harness evaluates how well large language models (LLMs) synthesize Java Modeling Language (JML) specifications by measuring behavioral adequacy across precondition and postcondition correctness and completeness. The study shows that while prompt optimization can raise verifier pass rates, many accepted specifications remain behaviorally weak, either over‑ or under‑constraining inputs and outputs. Spec‑Harness also serves as a feedback mechanism that improves the quality of specifications generated by general‑purpose coding agents and a specialized JML agent.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
4d ago

Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity

The paper introduces JAZ, a minimalist LLM agent framework that centers on a single primitive called “invoke”, which allows an LLM to write and execute arbitrary code, including recursive calls, while treating all inputs and interaction history as variables in the code environment. JAZ provides built‑in hooks for constraints and monitoring but relies solely on prompting, without external tools, memory systems, or file‑system access. Experiments show that JAZ “invoke” outperforms specialized external harnesses such as Letta (MemGPT) and ACE on long‑horizon recall tasks and continual self‑improvement, achieving higher accuracy at lower cost.

By Zhening Li, Joshua Liu, Mateja Vukelic, Nicole Shen, Supriya Lall, Amitayush Thakur, Alex Zhang, Omar Khattab, Jonathan Light, Armando Solar-Lezama
arXiv AI
Sep 21

SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?

arXiv:2609.21190v1 Announce Type: cross Abstract: Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check corr...

By George Ma, Benjamin Mikek, Haoyu Li, Ferhat Erata, Yuhao Zhang, Zeren Shui, Behrooz Omidvar Tehrani, Jun Huan, Murali Krishna Ramanathan, Somayeh Sojoudi, Hao Zhou, Anoop Deoras