Annotrace: A Workbench for Step-Level Annotation of Agent Traces

Annotrace is a workbench that converts agent traces from six formats—including OpenTelemetry GenAI, LangSmith, and Langfuse—into an ordered, nested list of execution steps. Users define a labelling codebook, and a frontier model proposes initial labels, enabling a joint human‑AI annotation process. The tool also measures agreement between annotators, reproducing a prior manual effort’s 235 annotation units and achieving 91% agreement (κ = 0.82) between two models on a multi‑agent question‑answering corpus.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents