← Back to all news
arXiv Computation and Language By Youngkyung Seo, MiHyeon Kim, Jisoo Lee, Hwijung Ryu

SERIAL-7: Deriving, Realizing, and Preserving Machine-Verifiable Outputs Across Representations

Read the original on arXiv Computation and Language →

SERIAL-7 is a Korean‑English benchmark that evaluates large language models on their ability to generate machine‑verifiable structured outputs across seven representations. It introduces an executable verification framework that separates semantic derivation (Derive), constrained artifact generation (Realize), and cross‑representation preservation (Preserve), using 177 validated artifacts and 43 atomic representation constraints. The benchmark shows that reliable structured generation requires explicit, executable verification of how artifacts are realized and preserved, as evidenced by varying compliance rates across ten models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Sep 15

Natural-Language to SysMLv2 Translation via Conformance-Driven Iterative Refinement

The paper introduces a framework that translates natural‑language descriptions into SysMLv2 models using a generate‑check‑repair loop driven by a SysMLv2 conformance checker. By embedding the checker as an oracle, the system iteratively repairs generated models until they achieve zero conformance errors, ensuring they are deployable in industrial modeling environments. Evaluation on 151 prompts across four large language models shows the approach raises production‑conformance acceptance from 51.16% to 100%.

By Chance LaVoie, Eladio Andujar Lugo, Taylan G. Topcu, Levent Burak Kara
llms
More like this →
arXiv Machine Learning
Sep 10

Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations

The paper evaluates how robust large language models are at generating SystemVerilog Assertions (SVA) when the underlying RTL code undergoes semantics‑preserving transformations such as operand reordering, identifier renaming, and redundant parenthesization. Using a curated dataset and two open‑source models (Qwen2.5‑Coder‑7B and DeepSeek‑Coder‑V2‑Lite), the authors find that 9.7%–27.0% of behaviors that were correct on the original RTL become incorrect after transformation, revealing significant instability that aggregate accuracy metrics can hide.

By FNU Aditi
llms
More like this →
arXiv AI
Oct 9

A Survey on LLM-Integrated Hardware Design Verification

The paper surveys how large language models (LLMs) are being used in hardware verification, covering tasks such as assertion generation, testbench creation, bug localization, and formal reasoning. It categorizes the literature by methodology, verification goal, tool interaction, benchmarks, and evaluation metrics, and distinguishes between inference-time techniques (prompting, retrieval, structured reasoning, agentic workflows) and training-time adaptation. The authors highlight that LLMs excel as semantic reasoning, search, and orchestration components within verification workflows, yet emphasize the need for semantic alignment, scalable integration, generalization, and rigorous evaluation to ensure true verification correctness.

By Hao Zheng, Jaime Rafael Imperial, Bardia Nadimi, Xiangfei Kong
llmsagentsbenchmarkssafety
More like this →