arXiv:2610.10513v1 Announce Type: cross Abstract: Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer...
arXiv:2610.18378v1 Announce Type: new Abstract: Applying process-based Earth science models to new sites and questions often requires expert calibration decisions. Reliable delegation to language-mod...
The paper introduces Belief-Calibrated Optimization (BCO), a method that records and updates a persistent in‑context document representing an agent’s belief about how the environment responds to edits. By continually revising this world model as new candidates are evaluated, BCO improves the performance of frozen LLM agents across five benchmarks, outperforming a control lacking the world model. An offline ablation shows that the document’s content provides reusable, accurate predictions of environmental responses, beyond mere form.
arXiv:2606.31422v3 Announce Type: replace Abstract: Long-horizon language agents enable autonomous decision-making in complex environments by planning and executing sequences of interdependent action...
Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both.
arXiv:2610.14381v1 Announce Type: new Abstract: Large language models are increasingly used as scientific assistants, yet we still lack reliable ways to evaluate whether they can investigate a system...
arXiv:2606. 11337v1 Announce Type: new Abstract: Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions.
arXiv:2607. 08046v1 Announce Type: cross Abstract: Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast.
arXiv:2609.07611v1 Announce Type: new Abstract: Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Exi...
The paper introduces a framework for evaluating large language model agents by attempting to end‑to‑end reproduce published astronomy studies, separating execution from verification and distinguishing computational failures from methodological ambiguities. Applying this to fourteen papers—one from The Astrophysical Journal and thirteen from Nature—revealed that eleven contained ambiguities that prevented a uniquely specified reproduction path. In a controlled case study, twelve different analysis paths produced distance estimates ranging from 2.16 to 3.53 kpc, with only one matching the published value of ~2.70 kpc, demonstrating that matching outcomes does not guarantee that the agent has reconstructed the underlying reasoning. whyItMatters":"The study shows that end‑to‑end reproduction can expose gaps in implicit scientific knowledge within AI systems, highlighting the need for better integration of causal relevance in LLM agents."
EvoBelief is a self‑evolving world model that represents environment knowledge as falsifiable hypotheses, allowing an LLM agent to update its beliefs during deployment. The system generates hypotheses when unexpected outcomes or unknowns arise, validates them through observations and controlled experiments, and then uses the confirmed rules to guide predictions or plans. Across benchmarks such as ALFWorld, MiniGrid, and DiscoveryWorld, EvoBelief improves task performance, revises stale rules, and discovers quantitative laws, outperforming baseline frozen world models.
The article discusses how AI foundation models trained on weather and climate data are increasingly fine‑tuned for a wide range of Earth science tasks beyond traditional weather forecasting. It highlights that current evaluations rely mainly on benchmark skill metrics, which assess forecast accuracy but not the models’ representation of underlying physical processes, a gap that becomes critical under changing climate conditions. The authors propose five priorities for physical evaluation—covering training data, fine‑tuning, behavioral testing, mechanistic interpretability, and output validation—and recommend three actions for the next decade: open AI‑ready evaluation datasets, a shared physics‑based reporting standard, and a dedicated research program on model safety.