← Back to all news
arXiv AI By Linjian Yang, Xinyan Wang, Kunpeng Liu

BoardSafety: AI-Ready Data from Manufacturer Documentation for Development-Board Safety

Read the original on arXiv AI →

The paper introduces BoardSafety, a benchmark that transforms manufacturer datasheets and hardware guides into a board‑specific safety rule base for evaluating large language models (LLMs) on hardware safety judgments. Each data item pairs a development board and a proposed operation with a safety rule and a ground‑truth label (safe, unsafe, or insufficient‑info). Experiments show that providing safety rules improves model accuracy, while structured‑rationale prompting yields mixed results, indicating that current models cannot reliably assess operation safety.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Sep 18

SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment

SAFARI is the first industrial benchmark for evaluating large language models (LLMs) in automotive hazard analysis and risk assessment (HARA) under ISO 26262. It comprises 3,000 de‑identified HARA cases and tests two tasks: open‑ended hazard generation and standards‑grounded risk classification, using a novel reference‑anchored LLM‑as‑a‑judge protocol. Experiments with nine state‑of‑the‑art LLMs show that while hazard narratives are often plausible, risk classification remains weak (best ASIL macro‑F1 = 0.261), with errors mainly due to missing scenario context and misjudged controllability. "whyItMatters":"The benchmark highlights the current limitations of LLMs in safety‑critical engineering workflows, guiding future research and expert oversight in automotive safety analysis."

By Chenxi Wu, Zimu Wang, Haiyang Zhang, Wei Wang, Zhijie Xu
llmsbenchmarkssafety
More like this →