This paper presents a black-box evaluation framework to systematically assess the ability of Large Language Models (LLMs) to generate Design Structure Matrices (DSMs) from structured technical documentation. Motivated by the closed-source nature of current Auto-DSM pipelines, the framework introduces a reproducible methodology that benchmarks generated DSMs (GEN-DSMs) against manually validated ground-truth matrices (GT-DSMs).
arXiv:2601. 17717v3 Announce Type: replace Abstract: Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities.
By Kaituo Zhang, Mingzhi Hu, Hoang Anh Duy Le, Fariha Kabir Torsha, Zhimeng Jiang, Minh Khai Bui, Chia-Yuan Chang, Yu-Neng Chuang, Zhen Xiong, Ying Lin, Guanchu Wang, Na Zou
arXiv:2602. 15983v3 Announce Type: replace-cross Abstract: Large language models (LLMs) can translate natural language into optimization code, but silent failures pose a critical risk: code that executes and returns solver-feasible solutions may encode semantically incorrect formulations---a feasibility--correctness gap reaching 90 percentage points on compositional problems.
By Junbo Jacob Lian, Yujun Sun, Huiling Chen, Chaoyu Zhang, Hanzhang Qin, Chung-Piaw Teo
The paper introduces a perturbation-based framework to evaluate quality metrics for large language model (LLM) generated Asset Administration Shells (AAS). By systematically degrading AAS outputs across multiple dimensions, the study identifies that exact property name matching and value-based recall, along with name-based F1 score, best reflect quality changes. Experiments on 6,400 AAS instances from 200 products across GPT‑4o‑mini, Qwen3, and DeepSeek‑R1 reveal how different perturbations affect metrics and highlight variations among model families and product segments.
By Janek Gro{\ss}, Elena Zentgraf, Jens Heidrich
arXiv:2607. 22759v1 Announce Type: cross Abstract: Large language models (LLMs) show promise in code generation, but their capabilities to produce correct, synthesizable hardware description language (HDL) code still remain to be properly benchmarked.
By Angshuman Chakravertty, Rahul Koshti, Buddhi Prakash Sharma, Vinay Chamola
arXiv:2609.37294v1 Announce Type: cross
Abstract: Large language models (LLMs) are increasingly used for software engineering tasks, yet their stochastic behavior challenges the validity, reproducibi...
By Francisco Ortin
arXiv:2601. 21787v2 Announce Type: replace-cross Abstract: The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions.
By Chantale Lauer, Peter Pfeiffer, Alexander Rombach, Nijat Mehdiyev
arXiv:2604. 05150v2 Announce Type: replace-cross Abstract: We study compiled AI, a paradigm in which large language models generate executable code artifacts during a compilation phase, after which workflows execute deterministically without further model invocation.
By Geert Trooskens (XY.AI Labs, Palo Alto, CA), Aaron Karlsberg (XY.AI Labs, Palo Alto, CA), Anmol Sharma (XY.AI Labs, Palo Alto, CA), Lamara De Brouwer (XY.AI Labs, Palo Alto, CA), Max Van Puyvelde (Stanford University School of Medicine, Stanford, CA), Matthew Young (XY.AI Labs, Palo Alto, CA), John Thickstun (Cornell University, Ithaca, NY), Gil Alterovitz (Brigham and Women's Hospital / Harvard Medical School, Boston, MA), Walter A. De Brouwer (Stanford University School of Medicine, Stanford, CA)
arXiv:2606. 05792v1 Announce Type: cross Abstract: TLA+ has supported industrial verification at companies such as Amazon and Microsoft, yet writing correct TLA+ specifications from natural language still requires time and expertise, which limits adoption.
By Arslan Bisharat, Brian Ortiz, Eric Spencer, Khushboo Bhadauria, TaiNing Wang, George K. Thiruvathukal, Konstantin Laufer, Mohammed Abuhamad
arXiv:2609.37405v1 Announce Type: cross
Abstract: Large language models (LLMs) are increasingly used for software engineering tasks that require understanding existing source code, including behavior...
By Ali Mohammadi Esfahani, Nafiseh Kahani, Samuel A. Ajila
arXiv:2608. 03311v1 Announce Type: cross Abstract: Enterprise compliance management requires rapid adaptation to evolving regulatory frameworks (e.
By Xavier Wrenn, Radoslav Raykov, Aleksandar Angelov, Hirokuni Kitahara, Yuji Watanabe, Anca Sailer
arXiv:2602.16715v2 Announce Type: replace
Abstract: We explore the potential of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), and Graph-based RAG (GraphRAG) for generating Desig...
By H. Sinan Bank, Daniel R. Herber