This is the opening piece of a four-part deep dive series, on building a high-frequency streaming pipeline against a live public API. The data source is openSenseMap, a citizen-science IoT network used for climate research, mostly in Germany.
The article discusses a matcher designed to clean up residual issues after normalization in a data lake. Testing revealed that no version of the matcher could be made fully safe, leading to its abandonment. The post then outlines the architecture that remained after the matcher was set aside.
arXiv:2607. 02731v1 Announce Type: cross Abstract: Machine learning has demonstrated significant potential for real-time monitoring, optimization, and control of scientific facilities.
A hands-on walkthrough of a hybrid local-cloud workflow using Gemma 4 and GPT-5. 4, with reasoning and structured outputs The post Stop Choosing Between Local and Cloud LLMs: A Field Guide to Hybrid Patterns appeared first on Towards Data Science .
arXiv:2607. 24532v1 Announce Type: new Abstract: Recent years have seen a rapid expansion in the production of large-scale geospatial maps derived from Earth observation (EO) data, driven largely by advances in machine learning (ML) and large computing infrastructure.
arXiv:2607. 08319v1 Announce Type: cross Abstract: We present GitLake, a Git-for-data design for an agent-first lakehouse.
arXiv:2512. 16455v4 Announce Type: replace-cross Abstract: The rapid growth of Artificial Intelligence and Machine Learning in scientific research has highlighted a gap between industry-standard MLOps tools and platforms, and the unique requirements of modern and Open Science, particularly regarding the FAIR (Findable, Accessible, Interoperable, and Reusable) principles.
The paper presents a carbon‑aware routing framework for function‑calling in large language models that distributes queries across a three‑tier edge‑cloud architecture. A lightweight k‑NN predictor estimates accuracy, delay, and power for each edge tier, and real‑time grid carbon intensity is used to route queries to the lowest‑emission tier that can execute them. Experiments on state‑of‑the‑art benchmarks show the framework matches cloud‑level accuracy while cutting operational carbon emissions by an average of four times.
arXiv:2606. 27342v1 Announce Type: cross Abstract: Entity Matching (EM) is a core operation in the data integration pipeline, where records from different sources are compared to determine whether they refer to the same real-world entity.
arXiv:2606. 15179v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) has emerged as a pivotal technique for improving language models by incorporating external knowledge at inference time.
The paper presents a systematic framework for large language model (LLM) watermarking as a provenance tool in big data ecosystems. It categorizes existing watermarking methods along four deployment dimensions—insertion point, verification authority, operational state, and transformation threat model—and aligns them with the big data principles of Volume, Velocity, Variety, Veracity, and Value. The authors introduce a readiness framework that maps four key workloads—online generation, streaming detection, transformation pipelines, and ecosystem governance—to system-level requirements such as throughput, false-positive control, robustness, cross-domain reliability, governance, and downstream utility, while highlighting gaps between benchmark performance and real-world deployment readiness.
arXiv:2606. 24113v1 Announce Type: new Abstract: Federated unlearning (FU) is critical for complying with legal mandates like the right to be forgotten in decentralized systems, yet current methods face a persistent dilemma between non-target knowledge loss and high request latency.
Starting with a local Parquet file, then joining it to data stored in the cloud The post Building a Data Lakehouse with DuckDB and DuckLake appeared first on Towards Data Science.