Annotrace is a workbench that converts agent traces from six formats—including OpenTelemetry GenAI, LangSmith, and Langfuse—into an ordered, nested list of execution steps. Users define a labelling codebook, and a frontier model proposes initial labels, enabling a joint human‑AI annotation process. The tool also measures agreement between annotators, reproducing a prior manual effort’s 235 annotation units and achieving 91% agreement (κ = 0.82) between two models on a multi‑agent question‑answering corpus.
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
arXiv:2609.24264v1 Announce Type: new Abstract: Tool-use agent traces identify messages and API calls, but procedural analyses also need explicit units of action and inspectable links to their eviden...
arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.
arXiv:2605. 21347v3 Announce Type: replace Abstract: Diagnosing failures in LLM agents remains largely manual.
arXiv:2610.08364v1 Announce Type: new Abstract: Frontier AI evaluations increasingly use open-ended, agentic, long-horizon tasks whose transcripts can span hundreds of pages of outputs and actions fr...
arXiv:2608. 06909v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly operate through long-horizon trajectories involving user instructions, tool use, external observations, and memory.
arXiv:2610.32415v1 Announce Type: new Abstract: LLM-based annotation has advanced rapidly, yet existing benchmarks remain misaligned with real-world practice. Actual annotation often involves lengthy...
arXiv:2606. 04990v1 Announce Type: cross Abstract: Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents.
AgentHop is a diagnostic benchmark comprising 1,011 multiple‑choice questions that test agentic multi‑hop question answering over scientific citation graphs. It provides a controlled seven‑tool sandbox with fixed token, turn, and tool‑call limits, allowing researchers to dissect model performance across four operational axes: retrieval, synthesis, tool‑call, and resource management. Analysis of 19 models shows distinct family‑specific behaviors, such as GPT models committing early, Anthropic and GLM checkpoints verifying before committing, DeepSeek and Kimi over‑searching, and Gemini‑3 Pro maintaining balance, while within‑family differences are revealed in retrieval versus synthesis strategies.
The paper investigates how a large language model can act as a corpus debugger for the Slovak portion of the multilingual FineWeb2 dataset. By using a tool‑using agent that explores the corpus via a read‑only, budgeted API, the authors identified seven issues across four phenomena, including a large machine‑generated content farm that slipped through the pipeline’s filters. The agent’s compiled test suite can scan the entire 26.5 M‑document corpus in 0.7 s without a GPU, providing precise prevalence estimates and actionable insights into pipeline failures.
The paper presents a novel approach to debugging multilingual web corpora by employing a large language model (LLM) as a tool-using agent rather than a classifier. Using this agent on the Slovak portion of FineWeb2, the authors identified seven distinct failure modes, including a large machine‑generated content farm that bypassed pipeline filters. The resulting test suite can scan the entire corpus in under a second without GPU resources, offering precise prevalence estimates and actionable insights into pipeline shortcomings.
arXiv:2606. 30560v1 Announce Type: cross Abstract: Coding agents are rapidly becoming a major application of agentic LLMs, but serving them efficiently remains challenging.