arXiv Computation and Language
RefinedGAW: A Refined German Academic Web Dataset
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
Hugging Face Trending Papers
Aug 19
Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale
Institutional Books – Enriched Text is a 2025 release that transforms Harvard Library’s 983,004-volume collection (IB‑HL) into a multilingual, annotated dataset. The pipeline normalizes OCR text while preserving metadata, separating endmatter, detecting paragraph language, clustering duplicates, and scoring bits‑per‑byte, all wrapped in HTML‑like annotations. The resulting IB‑HL‑ET contains 217 B tokens across 983,003 volumes and 1.39 B annotated subtopic paragraphs, enabling users to customize output rather than accept a single editorial decision.
arXiv AI
Aug 11
Guardian Crawler: Retrieval-First Knowledge Discovery with Bounded LLM Augmentation for Noisy Web Intelligence
arXiv:2608. 08994v1 Announce Type: cross Abstract: Retrieving relevant evidence from noisy web data is challenging, particularly in sensitive domains containing incomplete reports, heterogeneous language, and irrelevant content.
arXiv AI
Jul 28
CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
arXiv:2607. 22662v1 Announce Type: new Abstract: Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance.
arXiv AI
Oct 19
Optimist Deep Research: An Iterative, State-Centric Agent for Autonomous Literature Research
Optimist Deep Research is an open‑source agent that conducts literature research as an iterative, state‑centric process. It breaks a research question into a tree of subquestions, uses search intents to query multiple scholarly databases, and merges results into a provenance‑preserving document registry. The agent iteratively expands the registry via citation snowballing, evaluates evidence, and refines the research plan through a critic that can add, split, rephrase, deprioritize, or close subquestions, ultimately producing a report with every citation linked to a retrieved document.
arXiv AI
6d ago
DeGenTWeb: A First Look at MGT-dominant Websites
DeGenTWeb is a system that identifies websites whose informational prose is largely machine‑generated text (MGT) with minimal human input. By adapting MGT detectors for web pages and aggregating results across sites, the study reports that 20.9% of Common Crawl‑archived sites (Jan–Jul 2026) and 13.6% of Bing‑linked sites (2025) are MGT‑dominant, with a 0.12% mislabeling rate for human‑authored sites. The prevalence of MGT‑dominant sites is increasing over time and many share the same owner.
arXiv AI
Oct 9
RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees
RIT‑RAG is a retrieval‑augmented generation framework that combines content retrieval with structural navigation of documents. It builds a tree for each document from its table of contents or sitemap, then uses retrieved chunk positions to induce manageable sub‑trees that an LLM agent can navigate, read selectively, and reformulate queries. Experiments on financial, scientific, and customer‑support benchmarks show RIT‑RAG outperforms vanilla, graph‑based, and agentic baselines, achieving up to 11.4‑point accuracy gains on the new EntQABench of 2.84 million technical‑documentation webpages.