arXiv AI

Assessing the Business Process Modeling Competences of Large Language Models

arXiv:2601. 21787v2 Announce Type: replace-cross Abstract: The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions.

Hugging Face Trending Papers
Jul 7

Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation

This paper presents a black-box evaluation framework to systematically assess the ability of Large Language Models (LLMs) to generate Design Structure Matrices (DSMs) from structured technical documentation. Motivated by the closed-source nature of current Auto-DSM pipelines, the framework introduces a reproducible methodology that benchmarks generated DSMs (GEN-DSMs) against manually validated ground-truth matrices (GT-DSMs).

arXiv Machine Learning
Jun 5

Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation

arXiv:2606. 05792v1 Announce Type: cross Abstract: TLA+ has supported industrial verification at companies such as Amazon and Microsoft, yet writing correct TLA+ specifications from natural language still requires time and expertise, which limits adoption.

By Arslan Bisharat, Brian Ortiz, Eric Spencer, Khushboo Bhadauria, TaiNing Wang, George K. Thiruvathukal, Konstantin Laufer, Mohammed Abuhamad
arXiv AI
Sep 21

On the Limitations of Large Language Models for Conceptual Database Modeling

The article examines how Large Language Models can aid in creating Entity-Relationship diagrams from natural language requirements. It tests three LLMs with three prompting strategies—Zero-Shot, Chain of Thought, and Chain of Thought + Verifier—on scenarios of increasing complexity. Findings show that while LLMs perform adequately on simpler tasks, their reliability drops with more complex requirements, leading to inconsistencies, ambiguities, and constraint representation failures.

By Arthur F. Siqueira, Carlos D. S. Nogueira, Eduarda Farias, Claudio E. C. Campelo, J\'ulia Menezes
arXiv Computation and Language
Sep 22

LLJ Cards: Best practices for the Use of LLMs as Judges

arXiv:2609.24516v1 Announce Type: new Abstract: In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJs), these s...

By Khaoula Chehbouni, Melina Medjdoub, Florian Carichon, Golnoosh Farnadi, Jackie Chi Kit Cheung