arXiv AI
Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 7
Iris: Climbing to the Search Frontier
The paper introduces Iris-mini and Iris-pro, two search agents trained at 35B and 397B parameter scales. They use a novel data pipeline that constructs reverse‑engineered multi‑hop queries from web hyperlinks, filters trajectories, and alternates supervised fine‑tuning with reinforcement learning in a process called SFT‑RL climbing. Evaluations on several benchmarks show that, with inference‑time context management, the agents achieve the best open‑source results in their parameter ranges.
arXiv Machine Learning
Aug 18
GEO-Flag: Detecting and Measuring GEO-Optimized Web Content
arXiv:2608. 16824v1 Announce Type: new Abstract: Generative Engine Optimization (GEO) modifies web content to increase its likelihood of being selected and cited by generative search engines.
arXiv AI
Jul 28
CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
arXiv:2607. 22662v1 Announce Type: new Abstract: Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance.
arXiv AI
Aug 11
Guardian Crawler: Retrieval-First Knowledge Discovery with Bounded LLM Augmentation for Noisy Web Intelligence
arXiv:2608. 08994v1 Announce Type: cross Abstract: Retrieving relevant evidence from noisy web data is challenging, particularly in sensitive domains containing incomplete reports, heterogeneous language, and irrelevant content.
arXiv AI
Jun 11
Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality
arXiv:2606. 11499v1 Announce Type: cross Abstract: The performance of modern language models depends critically on pretraining data composition.
arXiv AI
Jun 8
Superintelligent Retrieval Agent: The Next Frontier of Agentic Retrieval
arXiv:2605. 06647v2 Announce Type: replace-cross Abstract: Retrieval-augmented agents are increasingly the interface to large knowledge bases, yet most treat retrieval as a black box: they issue exploratory queries, inspect snippets, and reformulate until evidence emerges.