The Flow has not summarised this story yet — read it at arXiv Machine Learning.
arXiv:2607. 16387v1 Announce Type: cross Abstract: An agent system's execution traces record how it fails, and procedures that improve such a system without changing model weights (trajectory selection, prompt and workflow optimization, runtime monitoring) read these traces for feedback.
The paper introduces a repeat‑run evaluation framework for AI agents that process large unstructured datasets, focusing on theme churn and volume disagreement as key metrics. Experiments on three customer‑feedback tasks across eight frontier models show that a taxonomy‑grounded agent (TGA) dramatically reduces theme churn by 86–88% and eliminates volume disagreement compared to raw generation and hierarchical decomposition. The framework is applicable beyond customer feedback, to any recurring synthesis of unstructured corpora such as financial reports, legal documents, incident records, and scientific literature.
AI agents are increasingly being programmed to automate knowledge work over large collections of unstructured data. Such automation requires repeatability: when the underlying evidence is unchanged, t...
arXiv:2608. 10970v1 Announce Type: cross Abstract: Recent advances in Large Language Models (LLMs) have demonstrated strong capabilities in generating semantically relevant concepts and relations, making them promising tools for taxonomy enrichment.
The paper investigates how different types of ontologies—curated, LLM-generated, and data-derived—serve as checkers for large language model agents answering questions over knowledge graphs. By evaluating these ontologies as prompt context, as checkers, or both, across both open-weight and proprietary agents on public and a large private graph, the study quantifies the trade-offs between errors caught and false alarms. It finds that curated ontologies can flag more correct answers than other sources but are only marginally more precise than random, and that combining sources or providing specific feedback offers limited benefits.
arXiv:2607. 28802v1 Announce Type: new Abstract: Existing evaluations often reduce agent failures to system-level outcomes, obscuring where the fault originated and which intervention would improve the agent system.