The article discusses a matcher designed to clean up residual issues after normalization in a data lake. Testing revealed that no version of the matcher could be made fully safe, leading to its abandonment. The post then outlines the architecture that remained after the matcher was set aside.
This is the opening piece of a four-part deep dive series, on building a high-frequency streaming pipeline against a live public API. The data source is openSenseMap, a citizen-science IoT network used for climate research, mostly in Germany.
arXiv:2607. 26298v1 Announce Type: new Abstract: We built and evaluated a self-serve entity resolution (ER) system on six benchmarks spanning 864 to 5M records, and three lessons emerged that are absent from existing ER literature.
arXiv:2606. 27342v1 Announce Type: cross Abstract: Entity Matching (EM) is a core operation in the data integration pipeline, where records from different sources are compared to determine whether they refer to the same real-world entity.
arXiv:2607. 18316v1 Announce Type: cross Abstract: Tool-augmented language-model agents execute multi-step workflows over external systems, resolving an entity once and then acting on it across subsequent steps.
Increasing context size in RAG systems doesn’t improve accuracy for aggregation tasks—it makes errors harder to detect. In this article, I benchmark retrieval-based pipelines against a deterministic full-scan engine across 100,000 rows and show why computation queries must be routed away from RAG entirely.
arXiv:2607. 10771v1 Announce Type: cross Abstract: Matching dependency is a generalization of the functional dependency concept, which allows users to apply custom similarity functions for matching individual attributes.
Enterprise Document Intelligence [Vol. 1 #7B] - Retrieval is filtering on structured tables: keywords first, TOC second, embeddings last The post Finding the right anchors for RAG: keyword, embedding, and TOC signals in parallel appeared first on Towards Data Science .
OpenSanctions Pairs is the first large‑scale public benchmark for entity matching on sanctions and OSINT data, comprising 755,540 expert‑labeled pairs drawn from over 1 million entities across 293 source datasets and 45 jurisdictions. The dataset spans multiple languages and writing systems, inconsistent structures, and time‑varying provenance, making it far more heterogeneous than prior benchmarks. Baseline experiments show a rule‑based matcher achieving 91.3 % F1, GPT‑4o reaching 99.0 % F1, and a locally deployable open‑source model scoring 98.2 % F1, with complementary failure modes that highlight the need to focus on downstream pipeline components.
Posted by Zilong Wang, Student Researcher, and Chen-Yu Lee, Research Scientist, Cloud AI Team People use tables every day to organize and interpret complex information in a structured, easily accessible format. Due to the ubiquity of such tables, reasoning over tabular data has long been a central topic in natural language processing (NLP).
Enterprise Document Intelligence [Vol. 1 #8A] - The schema is the contract: every field is a question the pipeline asks the model, and every answer is checkable The post Stop Returning Text from RAG: The Typed Answer Contract That Prevents Hallucination appeared first on Towards Data Science .
Enterprise Document Intelligence [Vol. 1 #11] - When the first answer points elsewhere in the document, the pipeline loops back to fetch the linked context The post Loop Engineering for Cross-References: When RAG Answers ‘see Section 7.
arXiv:2608.28622v1 Announce Type: cross Abstract: Large language model training corpora grow through successive, often redundant releases, so each release must be deduplicated against both itself and...