arXiv AI

SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

arXiv AI
Sep 2

Automated Event Log Generation from Unstructured Text Using Finetuned LLMs

The paper introduces a scalable framework that uses finetuned large language models (LLMs) to translate unstructured textual resources into structured event logs for process mining. By creating a new text-to-log dataset and finetuning LLMs on it, the authors demonstrate that the resulting models produce high‑fidelity event logs, outperforming few‑shot or zero‑shot prompting methods. This approach enables previously unused organizational data, such as incident tickets and manuals, to be incorporated into process mining workflows.

By Maximilian Seeth, Gabriel Marques Tavares, Daniel Schuster
arXiv AI
Aug 6

Traceable LLM-Generated Hazard Scenarios for Operational Safety Analysis of Aviation Systems Using ASRS Reports

arXiv:2608. 04697v1 Announce Type: new Abstract: Operational hazard analysis of aviation system operations must consider interactions among weather, ATC actions, airspace constraints, aircraft operations, and human factors - distinct from the functional hazard assessment applied at the aircraft-system level.

By Cristian Mascia, Roberto Pietrantuono, Daniel Rodriguez, Stefano Russo
arXiv AI
Aug 28

Learning to Predict, Discover, and Reason in High-Dimensional Event Sequences

The paper proposes a new framework for automated fault diagnostics in modern vehicles by treating diagnostic trouble codes (DTCs) as a high‑dimensional language. It introduces Transformer‑based models for predictive maintenance, scalable causal discovery methods, and a multi‑agent system that automatically generates Boolean error‑pattern rules. The approach aims to replace costly manual grouping of DTCs with scalable, data‑driven techniques.

By Hugo Math
arXiv Machine Learning
Aug 28

Stack Trace-Based Crash Deduplication with Transformer Adaptation

Stack Trace-Based Crash Deduplication with Transformer Adaptation introduces dedupT, a transformer‑based method that models entire stack traces instead of isolated frames. The approach first fine‑tunes a pretrained language model on stack traces and then trains a fully‑connected network to rank duplicate crashes. Experiments on four public datasets show dedupT improves Mean Reciprocal Rank by over 15% versus the best deep‑learning baseline and up to 10% over traditional methods, while also achieving higher ROC‑AUC for unique crash detection.

By Md Afif Al Mamun, Gias Uddin, Lan Xia, Longyu Zhang
arXiv AI
Sep 15

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

The paper introduces Continual Search, an iterative framework that guides large language models to persistently search for diagnostic evidence in long AI agent execution logs, addressing the limitations of one-shot judgments. Evaluated on four existing RCA benchmarks and a new large-scale dataset called MegaRCA-Mix, Continual Search consistently boosts attribution performance, achieving a 40% F1 improvement for GPT‑5.5 on MegaRCA‑Mix. The results show that effective search can outweigh raw model scale, enabling lower-tier models to outperform higher-tier ones in root‑cause attribution tasks.

By Harsh Raj, David Lee, Anas Mahmoud, Renxiong Wang, Razvan-Gabriel Dumitru, Chenguang Wang, Tong Zhao, Yunzhong He, Darvin Yi, Vipul Gupta
Hugging Face Trending Papers
Jul 30

From Single- to Cross-Document: Benchmarking Multi-Granularity Event Analysis of Large Language Models

Event analysis is an essential and fundamental direction of information extraction, involving various event-centric tasks at different granularity of documents. While large language models (LLMs) have preliminarily achieved promising performance in part of these tasks individually, their capability in event analysis still lacks comprehensive understanding due to restricted document granularity, task designs, and data source of existing benchmarks.

arXiv AI
1d ago

AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems

AuditKIT is a source‑available evaluation library that unifies benchmark accuracy, judge‑based assessment, runtime performance, tool‑use behavior, and retrieval quality into a single result representation. It offers direction‑aware metric contracts, configuration fingerprints, and extensible metrics and backends, while providing turn‑aware tool‑call matching and distinguishing verified task outcomes from trajectory‑level diagnostics. Seven case studies demonstrate how answer extraction, runtime behavior, and metric selection can alter evaluation outcomes across model comparison, serving runtimes, retrieval, tool calling, and interoperability.

By Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth
arXiv AI
Jun 9

RiskNet: A large-scale dataset of AI risk incidents from news with alignment and multi-dimensional annotations

arXiv:2606. 08376v1 Announce Type: cross Abstract: As artificial intelligence (AI) systems are increasingly deployed across socially consequential domains, reports of AI-related harms and failures have grown in frequency and diversity.

By Leihan Zhang, Wecheng Ye, Xianlong Ma, Haochuan Liu, Yang Li, Qianyu Zhang, Jinliang Chen, Qiang Yan