arXiv Machine Learning
SciExam for ENSO: Can AI Agents Build Climate Models?
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
Hugging Face Trending Papers
Oct 7
SciExam for ENSO: Can AI Agents Build Climate Models?
Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer, none of w...
arXiv AI
Oct 15
Delegating Earth science model calibration to knowledge-guided LLM agents
arXiv:2610.18378v1 Announce Type: new Abstract: Applying process-based Earth science models to new sites and questions often requires expert calibration decisions. Reliable delegation to language-mod...
arXiv AI
Sep 10
AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era
arXiv:2609.07611v1 Announce Type: new Abstract: Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Exi...
arXiv AI
Sep 3
Belief-Calibrated Optimization: An Explicit World Model for Agentic Optimization
The paper introduces Belief-Calibrated Optimization (BCO), a method that records and updates a persistent in‑context document representing an agent’s belief about how the environment responds to edits. By continually revising this world model as new candidates are evaluated, BCO improves the performance of frozen LLM agents across five benchmarks, outperforming a control lacking the world model. An offline ablation shows that the document’s content provides reusable, accurate predictions of environmental responses, beyond mere form.
arXiv AI
Sep 30
Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy
The paper introduces a framework for evaluating large language model agents by attempting to end‑to‑end reproduce published astronomy studies, separating execution from verification and distinguishing computational failures from methodological ambiguities. Applying this to fourteen papers—one from The Astrophysical Journal and thirteen from Nature—revealed that eleven contained ambiguities that prevented a uniquely specified reproduction path. In a controlled case study, twelve different analysis paths produced distance estimates ranging from 2.16 to 3.53 kpc, with only one matching the published value of ~2.70 kpc, demonstrating that matching outcomes does not guarantee that the agent has reconstructed the underlying reasoning. whyItMatters":"The study shows that end‑to‑end reproduction can expose gaps in implicit scientific knowledge within AI systems, highlighting the need for better integration of causal relevance in LLM agents."
arXiv AI
Jul 10
What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness
arXiv:2607. 08046v1 Announce Type: cross Abstract: Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast.