arXiv AI
Jul 3
The Agentic Garden of Forking Paths
arXiv:2607. 01507v1 Announce Type: new Abstract: Empirical research rarely admits a unique analysis.
AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable
arXiv:2606. 11456v1 Announce Type: cross Abstract: The deployment of LLM-based agents in scientific analysis raises opposing concerns: that agents may reduce methodological diversity, or that they may amplify the analytic flexibility through which researchers reach motivated conclusions.
Related stories
arXiv AI
Aug 21
Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis
arXiv:2608. 19902v1 Announce Type: new Abstract: AI agents can execute scientific analyses, but an analytic output becomes a defensible claim only after alternatives are weighed and the claim is limited to what the evidence supports.
arXiv AI
Jun 11
Preregistration for Experiments with AI Agents
arXiv:2606. 11217v1 Announce Type: cross Abstract: The proliferation of large language models (LLMs) and autonomous AI agents has given rise to a rapidly growing methodological paradigm: "in silico" behavioral experiments.
arXiv AI
Sep 1
HumanStudy-Bench: Towards AI Agent Design for Participant Simulation
arXiv:2602.00685v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as simulated participants in social science experiments, but their behavior is often unstable an...
arXiv AI
Jul 10
3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse
arXiv:2607. 07980v1 Announce Type: cross Abstract: Coding agents now author entire pull requests, and practitioners sharply disagree about what this does to code review: whether it becomes the bottleneck, whether human review is still necessary, and whether it quietly erodes the understanding that it once built.
arXiv AI
Aug 18
Position: AI Agents in Scientific Teams Should Be Studied as Human-Agent Systems
arXiv:2608. 14667v1 Announce Type: new Abstract: Large language model-based agents are increasingly deployed as collaborators in scientific discovery yet most current work focuses on the autonomous capabilities of "AI Scientists".
arXiv AI
Sep 7
TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents
TruthInsightBench is a new benchmark designed to evaluate automated scientific discovery agents by presenting them with 40 blind tasks drawn from peer‑reviewed studies across ten domains. Each task provides only a neutral objective and frozen data, withholding source conclusions, expected values, and analysis paths, forcing agents to determine which claim the data support. A fixed LLM‑based judge scores agents on evidentiary maturity across six dimensions, using 29 artifact‑grounded items, enabling fully automated, repeatable evaluation without human grading.
arXiv AI
Jun 8
Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle
arXiv:2606. 07462v1 Announce Type: new Abstract: As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution.
arXiv AI
Aug 7
Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap
arXiv:2608. 05179v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used across the scientific research lifecycle: ideation, literature search, experiment design and execution, analysis, manuscript drafting, and review.
arXiv AI
4d ago
Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results
arXiv:2604.21965v2 Announce Type: replace Abstract: Recent work has used LLM agents to reproduce empirical social science results with access to both the data and code. We broaden this scope by askin...
Hugging Face Trending Papers
Sep 21
SocioVerse2: A Longitudinal Dynamic Social Simulation Framework under a Human-AI Co-evolutionary Paradigm
Social simulation offers the social sciences an experimental instrument that the real world cannot supply, and generative agents have transformed it by acting as silicon samples that unite agent-based...
arXiv AI
Sep 3
Can Coding Agents Reproduce Findings in Computational Materials Science?
The paper introduces AutoMat, a benchmark designed to test large language model (LLM) coding agents on their ability to reproduce claims from computational materials science. AutoMat presents three challenges: reconstructing underspecified procedures, navigating specialized toolchains, and assessing whether the evidence supports a claim. Experiments show that current LLM agents achieve low success rates, with the best setting reaching only 53%, and failures stem mainly from incomplete procedures, methodological deviations, and execution fragility.