Towards Data Science
Jun 28
Tail Control: The Counterintuitive Engineering of Reliable Agentic Workflows
Behind a customer's API, a high-quality answer isn't enough. It has to be usable, which means on time.
How to Scale an Integration Pipeline Without Breaking Correctness
The article describes a real‑world case of scaling an enterprise integration pipeline from 500 to 8,000 events per second. It emphasizes that during this throughput increase, two correctness guarantees were strictly maintained and never compromised. The post illustrates how to achieve high performance while preserving essential data integrity constraints.
Related stories
Towards Data Science
Aug 13
Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model
Enterprise Document Intelligence [Vol. 1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right.
Towards Data Science
Sep 24
When the Correct Answer Is Nothing, What Does Your Pipeline Return?
The article discusses how the reliability mechanisms added to large language model (LLM) pipelines can lead to confident but incorrect outputs, especially when the correct answer is absent. It examines the behavior of pipelines in such scenarios and highlights the paradox where safeguards intended to improve accuracy may actually reinforce errors. The piece underscores the importance of understanding pipeline responses when faced with missing or ambiguous information.
Towards Data Science
Jun 19
I Tried to Schedule My ETL Pipeline. Here’s What I Didn’t Expect.
What I thought was a scheduling problem turned out to be a portability problem first The post I Tried to Schedule My ETL Pipeline. Here’s What I Didn’t Expect.
Towards Data Science
Jun 13
Larger Context Windows Don’t Fix RAG — So I Built a System That Does
Increasing context size in RAG systems doesn’t improve accuracy for aggregation tasks—it makes errors harder to detect. In this article, I benchmark retrieval-based pipelines against a deterministic full-scan engine across 100,000 rows and show why computation queries must be routed away from RAG entirely.
Towards Data Science
Jul 10
I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer
Building a production-ready RSS pipeline with Python, Docker, PostgreSQL, and Kestra The post I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer appeared first on Towards Data Science .
Towards Data Science
Jun 12
I Thought Data Engineering Was Just Writing Scripts. I Was Wrong.
I tried to make my ETL pipeline production-ready. Three things broke.
Towards Data Science
Jul 17
A Production RAG Pipeline in Action: Every Answer Typed and Cited
Enterprise Document Intelligence [Vol. 1 #9B] - One call wires the four upgraded bricks together, run on a paper, a NIST standard, and a report with a broken TOC The post A Production RAG Pipeline in Action: Every Answer Typed and Cited appeared first on Towards Data Science .
Towards Data Science
Aug 17
Loop Engineering for RAG: The Small Loops Inside Each Step, the Big Loops Across the Pipeline
Enterprise Document Intelligence [Vol. 1 #13bis] - The four bricks return useful results most of the time.
Towards Data Science
Aug 7
Loop Engineering for Listing Questions: When the Answer Is Every Passage, Not the Top One
Enterprise Document Intelligence [Vol. 1 #12] - The category of question most RAG pipelines silently fail on, and the pipeline shape that handles them The post Loop Engineering for Listing Questions: When the Answer Is Every Passage, Not the Top One appeared first on Towards Data Science .
Towards Data Science
Sep 22
Break Your Own RAG Pipeline Before Users Do
The article discusses a small adversarial test set designed to detect retrieval failures in Retrieval-Augmented Generation (RAG) pipelines that typical evaluation sets might miss. It emphasizes the importance of proactively testing your own RAG system to uncover hidden weaknesses before users encounter them. By using this targeted test set, developers can improve the reliability and robustness of their RAG models.
Towards Data Science
Jun 24
Your First Task as a Data Engineer in a New Company? Make the ETL Pipeline Testable
A practical data engineering onboarding workflow for environment setup, automated testing, and AI-assisted development. The post Your First Task as a Data Engineer in a New Company?