SAFARI is the first industrial benchmark for evaluating large language models (LLMs) in automotive hazard analysis and risk assessment (HARA) under ISO 26262. It comprises 3,000 de‑identified HARA cases and tests two tasks: open‑ended hazard generation and standards‑grounded risk classification, using a novel reference‑anchored LLM‑as‑a‑judge protocol. Experiments with nine state‑of‑the‑art LLMs show that while hazard narratives are often plausible, risk classification remains weak (best ASIL macro‑F1 = 0.261), with errors mainly due to missing scenario context and misjudged controllability.
"whyItMatters":"The benchmark highlights the current limitations of LLMs in safety‑critical engineering workflows, guiding future research and expert oversight in automotive safety analysis."
By Chenxi Wu, Zimu Wang, Haiyang Zhang, Wei Wang, Zhijie Xu
arXiv:2607. 17745v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly considered for environmental enforcement, but their ability to produce traceable enforcement decisions remains unclear.
By Ziliang Yang, Yi Zhang, Kaijun Lin, Jiachao Ke, Haihong Xu, Zongguo Wen
arXiv:2607. 01829v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants.
By Alex Brooker, Tim Hughes
The paper investigates whether existing AI safety benchmarks, designed for large language models, are suitable for evaluating small language models (SLMs). By testing five benchmark suites on 26 open‑source SLMs with a unified scoring rubric, the authors find that ambiguous judgments dominate, especially for complex prompts and certain architectures. This ambiguity, linked to factors like lexical density and output perplexity, undermines the reliability of aggregate leaderboards and reveals a confound between model capability and perceived safety.
By Nyamtulla Shaik, Fengjun Li, Bo Luo
arXiv:2508.15526v2 Announce Type: replace
Abstract: The rapid proliferation of large language models (LLMs) has intensified the requirement for reliable safety evaluation to uncover model vulnerabili...
By Xiangyang Zhu, Yuan Tian, Chunyi Li, Kaiwei Zhang, Wei Sun, Guangtao Zhai
IFC-MCQ is a method that automatically creates multiple-choice benchmarks from Industry Foundation Class (IFC) files, enabling exact option-set grading for large language model (LLM) queries in architecture, engineering, and construction (AEC). The taxonomy spans 15 categories and 36 subtypes across four groups, with answer keys derived from the IFC file and distractors generated from file-derived values or predefined rules. Experiments on IFC models of varying sizes and characteristics revealed category-specific failure modes and highlighted how performance can differ based on the source IFC model and retrieval interface.
By Rabindra Lamsal, Sisi Zlatanova, Xuesong Shen
The paper introduces DGEval, a benchmark of 1,678 questions designed to assess large language models (LLMs) on the International Maritime Dangerous Goods (IMDG) Code Amendment 42‑24. It evaluates 13 models from six providers, finding that while the best model surpasses human practitioners on multiple‑choice tasks, all models perform poorly on safety‑critical areas such as stowage, segregation, and regulatory recall. The study concludes that LLMs can aid compliance tasks—especially structured Dangerous Goods List lookups with web search—but human oversight and authoritative source verification remain essential for safety‑critical deployment.
By Alexander Thomas, Hubert P. H. Shum, Darren Nellis, Manli Zhu, Phatpicha Yochum, William Bartle, Daniel Wrightson
arXiv:2501. 14940v4 Announce Type: replace-cross Abstract: Aligning large language models (LLMs) with human values is essential for their safe deployment and widespread adoption.
By Guangzhi Sun, Xiao Zhan, Shutong Feng, Philip C. Woodland, Jose Such
The paper introduces BoardSafety, a benchmark that transforms manufacturer datasheets and hardware guides into a board‑specific safety rule base for evaluating large language models (LLMs) on hardware safety judgments. Each data item pairs a development board and a proposed operation with a safety rule and a ground‑truth label (safe, unsafe, or insufficient‑info). Experiments show that providing safety rules improves model accuracy, while structured‑rationale prompting yields mixed results, indicating that current models cannot reliably assess operation safety.
By Linjian Yang, Xinyan Wang, Kunpeng Liu
The paper "LLM Code Smells: A Taxonomy and Detection Approach" presents a refined taxonomy of nine LLM code smells and introduces SpecDetect4LLM, a static analysis tool for detecting them. Extensive empirical evaluation across 692 open‑source projects shows that 73.5% of systems contain LLM code smells, with the tool achieving 91.3% precision and 71.8% recall.
By Zacharie Chenail-Larcher, Brahim Mahmoudi, Naouel Moha, Quentin Sti\'evenart, Florent Avellaneda
arXiv:2606. 00160v1 Announce Type: cross Abstract: Large language models (LLMs) suffer from degraded safety capabilities even when fine-tuned with benign datasets.
By Junbo Zhang, Qianli Zhou, Xinyang Deng, Wen Jiang, Jie Pan, Jinbiao Zhu
arXiv:2606.09389v2 Announce Type: replace
Abstract: As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses...
By Yifan Chen, Haitao Li, Yiran Hu, Kaisong Song, Jun Lin, Yueyue Wu, Qingyao Ai, Min Zhang, Yiqun Liu