arXiv AI
Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 18
SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment
SAFARI is the first industrial benchmark for evaluating large language models (LLMs) in automotive hazard analysis and risk assessment (HARA) under ISO 26262. It comprises 3,000 de‑identified HARA cases and tests two tasks: open‑ended hazard generation and standards‑grounded risk classification, using a novel reference‑anchored LLM‑as‑a‑judge protocol. Experiments with nine state‑of‑the‑art LLMs show that while hazard narratives are often plausible, risk classification remains weak (best ASIL macro‑F1 = 0.261), with errors mainly due to missing scenario context and misjudged controllability. "whyItMatters":"The benchmark highlights the current limitations of LLMs in safety‑critical engineering workflows, guiding future research and expert oversight in automotive safety analysis."
arXiv AI
Jul 21
WuYu-EnvLE-Bench: A Benchmark for Evaluating Large Language Models in Environmental Law Enforcement
arXiv:2607. 17745v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly considered for environmental enforcement, but their ability to produce traceable enforcement decisions remains unclear.
arXiv AI
Jul 3
Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge
arXiv:2607. 01829v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants.
arXiv AI
Aug 19
Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models
The paper investigates whether existing AI safety benchmarks, designed for large language models, are suitable for evaluating small language models (SLMs). By testing five benchmark suites on 26 open‑source SLMs with a unified scoring rubric, the authors find that ambiguous judgments dominate, especially for complex prompts and certain architectures. This ambiguity, linked to factors like lexical density and output perplexity, undermines the reliability of aggregate leaderboards and reveals a confound between model capability and perceived safety.
arXiv Computation and Language
Sep 23
SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking
arXiv:2508.15526v2 Announce Type: replace Abstract: The rapid proliferation of large language models (LLMs) has intensified the requirement for reliable safety evaluation to uncover model vulnerabili...
arXiv Computation and Language
Oct 23
IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval
IFC-MCQ is a method that automatically creates multiple-choice benchmarks from Industry Foundation Class (IFC) files, enabling exact option-set grading for large language model (LLM) queries in architecture, engineering, and construction (AEC). The taxonomy spans 15 categories and 36 subtypes across four groups, with answer keys derived from the IFC file and distractors generated from file-derived values or predefined rules. Experiments on IFC models of varying sizes and characteristics revealed category-specific failure modes and highlighted how performance can differ based on the source IFC model and retrieval interface.