← Back to all news
arXiv AI By Yanis Merzouki, Coral Izquierdo, Matei Ignuta-Ciuncanu, Marcos Gomez-Bracamonte, Riccardo Maggioni, Alessandro Lombardi, Camilla Mazzoleni, Federico Martelli, Balazs Gunther, Jonas Petersen, Philipp Petersen

FactoryBench: Evaluating Industrial Machine Understanding

Read the original on arXiv AI →

FactoryBench is a benchmark designed to evaluate time‑series models and large language models on machine understanding using industrial robotic telemetry. It organizes Q&A pairs across four causal levels—state, intervention, counterfactual, and decision—based on Pearl’s ladder of causation, and includes five answer formats, with structured responses scored deterministically and free‑form responses evaluated by an LLM‑as‑judge voting protocol. The benchmark is built from over 70,000 Q&A items derived from roughly 15,000 episodes in the FactoryWave dataset (from a UR3 cobot and a KUKA KR10 arm) as well as AURSAD and voraus‑AD, and zero‑shot tests of six leading LLMs show that none surpass 50% on structured levels or 18% on decision‑making tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jun 10

LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.

By Haonan Wang, Jiaxiang Liu, Yurong Liu, Austin Senna Wijaya, Tianle Zhou, Eden Wu, Yijia Chen, Wanting You, Reya Vir, Daniela Pinto, Grace Fan, Yusen Zhang, Juliana Freire, Eugene Wu
llmsagentsnlpbenchmarks
More like this →
arXiv AI
Sep 3

UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents

UniToolCall introduces a unified framework for tool-use in large language model agents, standardizing toolset construction, dataset generation, and evaluation. The framework aggregates over 22,000 tools and creates a hybrid training corpus of more than 390,000 instances by combining ten public datasets with synthetically generated, structurally controlled trajectories. It models diverse interaction patterns—single‑hop vs. multi‑hop, single‑turn vs. multi‑turn, serial vs. parallel execution—and adds an Anchor Linkage mechanism to enforce cross‑turn dependencies, while converting seven public benchmarks into a common Query–Action–Observation–Answer format for fine‑grained evaluation.

By Yijuan Liang, Xinghao Chen, Yifan Ge, Ziyi Wu, Hao Wu, Changyu Zeng, Wei Xing, Xiaoyu Shen
llmsagentsfine-tuningbenchmarks
More like this →