arXiv:2608. 05179v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used across the scientific research lifecycle: ideation, literature search, experiment design and execution, analysis, manuscript drafting, and review.
The paper introduces an end‑to‑end pipeline that harvests, validates, and models links between scholarly articles and their source code from journals such as JOSS, SoftwareX, and IPOL, as well as SIGMOD ARI reproducibility reports. It produces a curated set of 4,397 DOI‑repository pairs and defines two Wikidata‑based application profiles—one for articles and one for software—aligned with schema.org and CodeMeta. Using these profiles, the authors created 4,182 new Wikidata software items linked to their papers, while only 82 repositories were previously represented, and they demonstrate compatibility with the COAR Notify protocol for future live enrichment.
OpenAI4S is an open‑source scientific research agent that treats code as action and science as sessions, combining a persistent computing runtime with structured session management. It uses tool calls for orchestration, executes code cells in persistent Python and R kernels, and records an append‑only Action Ledger, per‑cell execution logs, versioned artifacts, environment snapshots, and workspace checkpoints to preserve provenance and enable session recovery, branching, and extension. Evaluated on 36 research scenarios—including retrosynthesis, molecular dynamics, and protein design—OpenAI4S achieved a higher overall score (7.83) than a general‑purpose coding harness, especially on long‑horizon, computation‑intensive workflows, though reproducibility remains an open challenge. whyItMatters":"The system demonstrates that persistent execution coupled with session‑level provenance can enhance the reliability of AI‑assisted scientific workflows, as evidenced by its superior performance across diverse research scenarios."
StagedWorkspace is a versioned workspace designed for knowledge‑work AI agents, ensuring that every parsed view, native file edit, and review diff is explicitly tied to a specific version of the workspace state. By binding parsed records and review diffs to content hashes of native files, the system improves performance on tasks such as OfficeQA and APEX‑Agents, achieving higher pass rates and rubric scores compared to single‑view approaches. The study demonstrates that providing dual parsed/native access and visible diffs enhances agent performance, highlighting workspace state as a key experimental variable for future benchmarks.
ReproAgent is a four‑stage pipeline—Prepare, Plan, Generate, Repair—that uses a persistent implementation contract to guide scientific AI agents in converting research papers into executable code repositories. The system employs two channels: an implementation‑requirement channel that translates paper snippets into code obligations, and a reference‑evidence channel that pulls content and structure from related repositories. Evaluated on PaperBench Code‑Dev, ReproAgent achieves the highest mean score among same‑backbone scaffolds for both Claude‑Sonnet‑4.5 and Gemini‑3‑Flash, with ablation studies confirming the contribution of both channels.
CrossAudit proposes a Git‑native, cross‑vendor audit protocol for autonomous research pipelines, ensuring each work increment is reviewed by an agent from a different vendor against a human‑written rulebook. Audit outcomes, disputes, and rulings are stored as git commits, providing a replayable, versioned supervision history. The authors implemented the protocol with GitHub Actions and Python, deployed it in a computational‑chemistry pipeline, and conducted a seeded‑defect trial that revealed differing interpretations of the same rulebook by two vendors.
arXiv:2607. 02520v1 Announce Type: cross Abstract: Automated research agents increasingly generate code, retrieve literature, and draft scientific artifacts, but they often fail to verify whether generated experiments execute correctly or whether cited sources support generated claims.
The paper introduces a six‑stage audit framework for assessing reproducibility in computer science literature and applies it to the neuro‑symbolic AI (NSAI) subfield. Using the framework, the authors screened 5,497 records, identified 1,304 eligible studies, and found verifiable code artifacts for only 455 of them. Of those, they fully or partially reproduced 85 studies, representing 6.52% of the eligible corpus and 18.68% of attempted reruns, highlighting a significant reproducibility gap even when code is declared available.
The study surveys popular, active GitHub repositories across thirteen languages, treating each repository’s AGENTS.md and CLAUDE.md files as instruction sets for coding agents. It finds that most repositories include such instructions, with CLAUDE.md often referencing AGENTS.md, and that these files contain a mix of project knowledge and workflow rules, including varying stances on AI attribution. The analysis reveals that more popular projects tend to have stricter pull‑request rules and AI contribution guidelines, and that LLM and agent projects are over‑represented among adopters.
K‑Dense BYOK is a free, open‑source AI research assistant that runs locally on a researcher’s own computer. It provides a structured environment with scientific procedures, workflow templates, and a living lab notebook that logs all actions without allowing the agent to alter the record. The system emphasizes reproducibility by recording the software environment and offering commands to regenerate results, outperforming managed platforms on interdisciplinary research prompts.
SpecMine is a large-scale corpus that documents Spec-Driven Development (SDD) artifacts in public GitHub repositories. It includes a broad census of 470,795 spec files from 73,030 repositories linked to 17 tools, a focused census of 98,574 Kiro layout files from 12,910 repositories, and a sweep of 5,992 pull requests across 581 repositories that modify specs. The dataset provides enriched metadata, full commit histories, parsed document structures, and over 2.4 million typed references connecting specs to code, sibling documents, PRs, branches, and issues.
LabFactory is a framework that transforms a scientific brief into an executable AI lab, integrating models, knowledge resources, tools, and a controller behind a fixed interface. The builder packages the lab in a metered workspace, and a separate host evaluates the delivered artifact on held‑out inputs, ensuring the system itself is the evaluation target. Across 28 constructions in seven scientific domains, the delivered labs surpassed reference values on all 33 subtests, demonstrating that an AI agent can fully realize a scientific brief into a working, inspectable lab.