arXiv Machine Learning

Not All Invariants Are Equal: Curating Training Data to Accelerate Program Verification with SLMs

arXiv:2603. 15510v2 Announce Type: replace Abstract: The synthesis of inductive loop invariants remains a critical bottleneck in automated program verification.

arXiv AI

Aug 11

Quokka: Accelerating Program Verification with LLMs via Invariant Synthesis

arXiv:2509. 21629v4 Announce Type: replace-cross Abstract: Program verification relies on loop invariants, yet automatically discovering strong invariants remains a long-standing challenge.

By Anjiang Wei, Tianran Sun, Tarun Suresh, Haoze Wu, Ke Wang, Alex Aiken

llms fine-tuning benchmarks

arXiv AI

Jun 15

HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification

arXiv:2606. 13706v1 Announce Type: cross Abstract: We present HierSVA, an integrated suite that combines a pipeline, dataset, and benchmark for LLM-driven hierarchical hardware formal verification.

By Maohua Nie, Jiang Zhu, Jingqun Zhang, Zhichen Zeng, Jiayi Wang, Sibo Zhang, Jialin Wang, C. -J. Richard Shi

llms agents benchmarks

arXiv AI

Jun 4

AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms

arXiv:2602. 09464v2 Announce Type: replace-cross Abstract: Vericoding refers to the generation of formally verified code from rigorous specifications.

By Haoyu Zhao, Ziran Yang, Jiawei Li, Deyuan He, Zenan Li, Chi Jin, Venugopal V. Veeravalli, Aarti Gupta, Sanjeev Arora

llms benchmarks

arXiv Machine Learning

Jul 10

Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation

arXiv:2607. 07748v1 Announce Type: new Abstract: Large Language Models achieve strong code generation for high resource languages like Python and Java but suffer sharp performance drops on Low-Resource Programming Languages~(LRPLs) such as Julia.

By Didula Samaraweera, Anjana Supun, Srinath Perera

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 2

T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Models

arXiv:2504. 04718v2 Announce Type: replace-cross Abstract: Recent studies have demonstrated that test-time compute scaling effectively improves the performance of small language models (sLMs).

By Minki Kang, Jongwon Jeong, Jaewoong Cho

llms efficiency benchmarks

arXiv AI

Jun 16

LLM4RTL: Tool-Assisted LLM for RTL Generation

arXiv:2606. 15500v1 Announce Type: cross Abstract: Large language models (LLMs) have facilitated impressive progress in software engineering, code generation, tooling, and systems.

By Jing Jin, Robert Chu, Ning Yan, Masood S. Mortazavi

llms fine-tuning benchmarks

arXiv AI

Aug 11

P$^{3}$: Joint Program-and-Proof Planning for Verified Code Generation

arXiv:2608. 09277v1 Announce Type: new Abstract: Verified code generation asks a large language model (LLM) to generate both an executable program and a machine-checkable proof that the program meets a formal specification, promising software that is correct by construction.

By Zenan Li, Ziran Yang, Peiyang Song, Zhaoyu Li, Kaiyu Yang

llms agents benchmarks

arXiv AI

Jul 23

ITPEval: Benchmarking Formal Translation Across Interactive Theorem Provers

arXiv:2607. 19407v1 Announce Type: new Abstract: Formal theorem proving has emerged as a frontier challenge for machine learning, yet the ecosystem is fragmented: proofs remain siloed across incompatible systems, limiting both training data for learning-based provers and the portability of verified results.

By Jiayi Wu, Robert Joseph George, Anima Anandkumar

llms benchmarks

arXiv AI

Jul 28

TLA$^{+}$-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA+ Specification Generation

arXiv:2607. 23425v1 Announce Type: cross Abstract: Large language models increasingly write TLA$^{+}$ formal specifications from natural-language descriptions, but progress is hard to measure: existing resources grade by resemblance to a reference or by whether the output parses, neither of which shows correctness.

By Arslan Bisharat, Eric Spencer, Brian Ortiz, Khushboo Bhadauria, Mujtaba Nazari, Beatriz Santos, Anisa Ramos, TaiNing Wang, George K. Thiruvathukal, Konstantin L\"aufer, Mohammed Abuhamad

llms benchmarks

arXiv Machine Learning

Jul 31

LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints

arXiv:2410. 06458v2 Announce Type: replace-cross Abstract: Instruction following is a key capability for LLMs.

By Thomas Palmeira Ferraz, Kartik Mehta, Yu-Hsiang Lin, Haw-Shiuan Chang, Shereen Oraby, Sijia Liu, Vivek Subramanian, Tagyoung Chung, Mohit Bansal, Nanyun Peng

llms benchmarks

arXiv Machine Learning

Jul 28

Benchmarking LLMs for Verilog Design Flows

arXiv:2607. 22759v1 Announce Type: cross Abstract: Large language models (LLMs) show promise in code generation, but their capabilities to produce correct, synthesizable hardware description language (HDL) code still remain to be properly benchmarked.

By Angshuman Chakravertty, Rahul Koshti, Buddhi Prakash Sharma, Vinay Chamola

llms benchmarks

arXiv Machine Learning

Jun 5

Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation

arXiv:2606. 05792v1 Announce Type: cross Abstract: TLA+ has supported industrial verification at companies such as Amazon and Microsoft, yet writing correct TLA+ specifications from natural language still requires time and expertise, which limits adoption.

By Arslan Bisharat, Brian Ortiz, Eric Spencer, Khushboo Bhadauria, TaiNing Wang, George K. Thiruvathukal, Konstantin Laufer, Mohammed Abuhamad

llms safety

Related stories