OpenAI Blog
Apr 2, 2025
PaperBench: Evaluating AI’s Ability to Replicate AI Research
We introduce PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research.
The Secret to Reproducible and Portable Optimization: ORPilot’s Intermediate Representation (IR)
Why production-level AI optimization modeling agent needs reproducibility and portability, and how IR helps achieve them The post The Secret to Reproducible and Portable Optimization: ORPilot’s Intermediate Representation (IR) appeared first on Towards Data Science .
Related stories
arXiv AI
Jun 4
SciDER: Scientific Data-centric End-to-end Researcher
arXiv:2603. 01421v3 Announce Type: replace Abstract: While large language models accelerate scientific discovery, existing agents face severe limitations in adaptability, domain generalization, and multimodal scalability, often struggling to autonomously process raw, domain-specific experimental data.
arXiv AI
Jun 18
Externalizing Research Synthesis and Validation in AI Scientists through a Research Harness
arXiv:2606. 18874v1 Announce Type: new Abstract: AI systems can increasingly automate scientific workflows, but the reasoning that links prior evidence, generated ideas, experiments and final claims often remains implicit inside model inference.
arXiv AI
Aug 11
Rethinking Self-Evolving Agents: Do We Still Need Prescribed Optimization Pipelines?
arXiv:2608. 09629v1 Announce Type: new Abstract: Self-evolving agents are usually built around prescribed optimization pipelines: the framework decides how to gather evidence, revise a persistent artifact, select candidates, and stop.
arXiv AI
Jun 26
AgentX: Towards Agent-Driven Self-Iteration of Industrial Recommender Systems
arXiv:2606. 26859v1 Announce Type: new Abstract: Recommendation algorithm iteration is moving from an artisanal, engineer-bound process toward an industrialized research loop, but this transition remains blocked by a structural execution bottleneck: the idea-to-launch cycle still depends on human engineers to generate hypotheses, modify production code, launch A/B experiments, and attribute online results.
arXiv AI
Jul 20
Recursive Harness Self-Improvement
arXiv:2607. 15524v1 Announce Type: cross Abstract: Under model--harness co-evolution, harnesses are not merely inference-time scaffolds but data-generating components whose execution traces can shape future foundation models.
arXiv AI
Aug 6
The RAIL Principles for Neurosymbolic AI: Reasoning, Assurances, Interfacing and Learning
arXiv:2608. 04285v1 Announce Type: new Abstract: Neurosymbolic AI systems that integrate machine learning and symbolic reasoning are rapidly gaining attention.
arXiv AI
Aug 6
A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination
arXiv:2608. 04872v1 Announce Type: cross Abstract: Symbolic regression aims to discover closed-form equations from data, but existing LLM-guided methods often rely on a unified proposal loop that compresses heterogeneous search failures into a scalar score and a single prompt.
arXiv AI
Jul 9
Power and Limitations of Aggregation in Compound AI Systems
arXiv:2602. 21556v2 Announce Type: replace Abstract: When designing compound AI systems, a common approach is to query multiple copies of the same model and aggregate the responses to produce a synthesized output.
arXiv AI
Jun 12
From Digital to Physical: Digital Agents as Autonomous Coaches for Physical Intelligence
arXiv:2601. 21570v2 Announce Type: replace Abstract: The field of Embodied AI is witnessing a rapid evolution toward general-purpose robotic systems, fueled by high-fidelity simulation and large-scale data collection.
arXiv AI
Aug 3
AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
arXiv:2607. 29626v1 Announce Type: new Abstract: As LLMs evolve from code completion systems into autonomous scientific agents, evaluating their ability to conduct experiments has become increasingly important.
arXiv AI
Jul 21
From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Data
arXiv:2607. 16845v1 Announce Type: new Abstract: Scientists at European XFEL conduct experiments that generate very large and complex datasets.