arXiv Machine Learning
1d ago
WANDR: A Benchmark for Wide and Deep Research
arXiv:2608. 14747v1 Announce Type: new Abstract: WANDR (Wide ANd Deep Research) is a benchmark of 500 realistic, challenging data-collection tasks for research agents.
Industry Classification of GitHub Repositories Using the North American Industry Classification System (NAICS)
arXiv:2607. 06505v1 Announce Type: cross Abstract: GitHub hosts hundreds of millions of public repositories, but the platform exposes no native mapping from repositories to standardized industry sectors.
Related stories
arXiv AI
Jul 29
Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
arXiv:2607. 25891v1 Announce Type: new Abstract: Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules.
arXiv AI
Jun 3
The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection
arXiv:2606. 03305v1 Announce Type: new Abstract: Benchmark contamination, where evaluation examples appear in a model's training data, threatens the validity of LLM assessment.
arXiv AI
Jun 9
GIScholarBench: Benchmarking LLM Overconfidence in GIS Research
arXiv:2606. 08036v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used in academic research workflows, but scholarly tasks require high factual precision and therefore expose a key weakness: overconfidence.
arXiv AI
Jul 8
Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmark
arXiv:2607. 05443v1 Announce Type: cross Abstract: Scientists increasingly rely on open-source tools to support their research workflows, yet discovering relevant software among over 600 million GitHub repositories remains challenging.
arXiv AI
Jul 29
Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
arXiv:2607. 24882v1 Announce Type: cross Abstract: Modern coding agents are usually evaluated by whether they eventually produce a correct patch, but patch generation depends on an earlier context-acquisition stage: finding the repository files needed for the task.
arXiv AI
Jun 24
Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories
arXiv:2606. 24429v1 Announce Type: cross Abstract: Generative AI coding agents are entering the open-source supply chain, yet their diverse and often invisible traces leave their prevalence poorly understood.
arXiv Machine Learning
Jul 24
DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
arXiv:2607. 20465v1 Announce Type: new Abstract: The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end.
arXiv AI
Aug 6
CIDR: A Large-Scale Industrial Source Code Dataset for Software Engineering Research
arXiv:2605. 12153v2 Announce Type: replace-cross Abstract: We present the Curated Industrial Developer Repository (CIDR), a large-scale dataset of real-world software repositories collected from industrial partners.
arXiv Machine Learning
Jun 30
KrishokChat: A Citation-Grounded Dataset and Benchmark for Bengali Agricultural Advisory
arXiv:2606. 29243v1 Announce Type: new Abstract: We present KrishokChat, the first citation-grounded Bengali agricultural instruction-tuning dataset for crop advisory in low-resource settings.
Hugging Face Trending Papers
Jun 23
Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories
Generative AI coding agents are entering the open-source supply chain, yet their diverse and often invisible traces leave their prevalence poorly understood. We introduce a multi-layered detection framework that integrates configuration-file scanning, commit-message analysis, author-identity matching, and bot-signature lookup across World of Code (180M+ Git repositories), classifying agent traces into four behavioral types.
arXiv AI
Jul 1
RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora
arXiv:2604. 19047v2 Announce Type: replace-cross Abstract: Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate on corpora such as financial reports, legal codes, and patents, where information is highly redundant and documents exhibit strong inter-document similarity.