What I thought was a scheduling problem turned out to be a portability problem first The post I Tried to Schedule My ETL Pipeline. Here’s What I Didn’t Expect.
Building a production-ready RSS pipeline with Python, Docker, PostgreSQL, and Kestra The post I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer appeared first on Towards Data Science .
A practical data engineering onboarding workflow for environment setup, automated testing, and AI-assisted development. The post Your First Task as a Data Engineer in a New Company?
I tried to make my ETL pipeline production-ready. Three things broke.
A practical walkthrough using text-to-SQL as the example The post Why I Stopped Using One Agent and Built a Multi-Agent Pipeline Instead appeared first on Towards Data Science .
The article describes a real‑world case of scaling an enterprise integration pipeline from 500 to 8,000 events per second. It emphasizes that during this throughput increase, two correctness guarantees were strictly maintained and never compromised. The post illustrates how to achieve high performance while preserving essential data integrity constraints.
The article discusses how the reliability mechanisms added to large language model (LLM) pipelines can lead to confident but incorrect outputs, especially when the correct answer is absent. It examines the behavior of pipelines in such scenarios and highlights the paradox where safeguards intended to improve accuracy may actually reinforce errors. The piece underscores the importance of understanding pipeline responses when faced with missing or ambiguous information.
EVE Online is beginning its transition to Python 3, a move that will involve using the futurize script on 2.4 million lines of code and a manual review of about 20,000 differences between Python 2 and Python 3. The company has historically run on Stackless Python since 2003, with the last major upgrade in 2010 to Stackless Python 2.7. While the announcement does not detail how Stackless will be replaced, the team previously showcased a shift away from Stackless in their Carbon engine for EVE Frontier, leveraging the open‑source carbonengine/scheduler library.
The article recounts the author's experience of moving a Dockerized data pipeline from a local laptop to AWS, highlighting the challenges that arose when the environment changed. It explores lessons learned about container behavior, networking intricacies, and hidden assumptions that were previously taken for granted in a local setup. The post serves as a practical guide for developers facing similar transitions to cloud infrastructure.
A hands-on guide to tracking experiments, logging models, and reproducing results with ML Flow. The post Are Your ML Experiments a Mess?
How AI has massively changed my day-to-day workflow The post A Day in the Life of a Data Scientist in 2026 appeared first on Towards Data Science .
Follow this framework to build a project that will impress hiring managers The post The Exact ML Project I’d Build to Get Hired in 2026 appeared first on Towards Data Science .
Enterprise Document Intelligence [Vol. 1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right.