FactoryBench is a benchmark designed to evaluate time‑series models and large language models on machine understanding using industrial robotic telemetry. It organizes Q&A pairs across four causal levels—state, intervention, counterfactual, and decision—based on Pearl’s ladder of causation, and includes five answer formats, with structured responses scored deterministically and free‑form responses evaluated by an LLM‑as‑judge voting protocol. The benchmark is built from over 70,000 Q&A items derived from roughly 15,000 episodes in the FactoryWave dataset (from a UR3 cobot and a KUKA KR10 arm) as well as AURSAD and voraus‑AD, and zero‑shot tests of six leading LLMs show that none surpass 50% on structured levels or 18% on decision‑making tasks.
arXiv:2609.14142v1 Announce Type: new Abstract: Large language models (LLMs) can struggle with time-series question answering (TS-QA), especially when numerical signals are serialized as text and req...
arXiv:2609.35879v1 Announce Type: cross Abstract: Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many com...
arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.
UniToolCall introduces a unified framework for tool-use in large language model agents, standardizing toolset construction, dataset generation, and evaluation. The framework aggregates over 22,000 tools and creates a hybrid training corpus of more than 390,000 instances by combining ten public datasets with synthetically generated, structurally controlled trajectories. It models diverse interaction patterns—single‑hop vs. multi‑hop, single‑turn vs. multi‑turn, serial vs. parallel execution—and adds an Anchor Linkage mechanism to enforce cross‑turn dependencies, while converting seven public benchmarks into a common Query–Action–Observation–Answer format for fine‑grained evaluation.
arXiv:2606. 15107v1 Announce Type: new Abstract: Time series data in real-world deployments is overwhelmingly irregular.
arXiv:2610.15228v1 Announce Type: new Abstract: Words are increasingly becoming an interface for action: large language models (LLMs) can now invoke tools, operate devices, and make decisions that af...
arXiv:2609.13457v1 Announce Type: new Abstract: Timeseries multimodal large language models (TS-MLLMs) have recently begun leveraging the reasoning capabilities of large language models (LLMs) for qu...
arXiv:2606. 01498v1 Announce Type: cross Abstract: Time series data inform critical decisions across many real-world domains.
arXiv:2605. 09081v4 Announce Type: replace-cross Abstract: We introduce the first universal pretraining corpus for industrial time-series data: FactoryNet.
arXiv:2608. 16224v1 Announce Type: cross Abstract: By leveraging large-scale pretraining, LLMs can interpret diverse temporal expressions and question formulations without task-specific training.
The paper introduces CRAFT, a data‑centric fine‑tuning approach that aligns small language models (SLMs) for pre‑hoc reasoning in AI‑native 6G radio access networks (RAN). By automatically generating verified (input, trace, label) triplets and fine‑tuning with low‑rank adaptation, CRAFT achieves high accuracy and F1 scores on TRACTOR and IC xApp datasets while avoiding parse failures that plague RL methods like GRPO. It also reduces energy consumption by 59% compared to GRPO baselines, offering a more sustainable path to auditable AI in 6G RAN.
Identifying root causes in production microservice failures requires reasoning over large-scale, multimodal telemetry spanning metrics, logs, and traces, a problem that has proved resistant to both classical and LLM-based approaches. The OpenRCA dataset exemplifies these challenges: it is large-scale, multimodal, and lacks detailed domain knowledge, and yields consistently low accuracy across all existing methods.