The Flow has not summarised this story yet — read it at arXiv Computation and Language.
In so-called "Industry 4.0," integrated sensors, analytics tools, and other digital technologies supplement industrial workflows, potentiating enhanced visibility and automation. Prognostics and healt...
DiagnosticIQ is a benchmark that tests large language models (LLMs) on translating engineer‑written symbolic rules into maintenance actions for industrial assets. It contains 6,690 multiple‑choice questions derived from 118 expert rule‑action pairs across 16 asset types, and includes five variants that probe different failure modes. The benchmark evaluates 29 LLMs and four embedding baselines, showing that while top models perform near the frontier, they exhibit brittleness under distractor expansion and structural perturbations.
ToolRobustBench is a stage-wise diagnostic benchmark designed to evaluate and diagnose failures in tool‑calling agents, which are large language models that select tools, provide structured arguments, and interpret tool feedback. The benchmark aligns four perturbation families—tool‑interface, user‑intent, tool‑output/observation, and runtime‑environment—with the tool‑use pipeline, attributing failures to specific stages such as tool selection, schema grounding, argument binding, and feedback handling. Experiments across 15,456 instances, 7 models, and 16 local tools reveal that while overall performance is high, robustness degrades significantly, especially under tool‑output/observation perturbations, and mixed‑family perturbations produce non‑additive failure patterns.
arXiv:2606. 17904v1 Announce Type: new Abstract: Language models increasingly serve as advisory systems in maintenance operations.
arXiv:2603. 28590v3 Announce Type: replace Abstract: Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs.
arXiv:2607. 04686v1 Announce Type: cross Abstract: Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails.