arXiv:2606. 13148v1 Announce Type: new Abstract: Climate and environmental decision-making increasingly requires reasoning across heterogeneous inputs, including gridded physical data, satellite imagery, geospatial context, and simulator outputs.
The paper introduces a framework for evaluating large language model agents by attempting to end‑to‑end reproduce published astronomy studies, separating execution from verification and distinguishing computational failures from methodological ambiguities. Applying this to fourteen papers—one from The Astrophysical Journal and thirteen from Nature—revealed that eleven contained ambiguities that prevented a uniquely specified reproduction path. In a controlled case study, twelve different analysis paths produced distance estimates ranging from 2.16 to 3.53 kpc, with only one matching the published value of ~2.70 kpc, demonstrating that matching outcomes does not guarantee that the agent has reconstructed the underlying reasoning. whyItMatters":"The study shows that end‑to‑end reproduction can expose gaps in implicit scientific knowledge within AI systems, highlighting the need for better integration of causal relevance in LLM agents."
arXiv:2608.23525v1 Announce Type: new Abstract: Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality. Natural hazards ma...
arXiv:2609.12533v1 Announce Type: cross Abstract: Real-world Earth observation (EO) agents must translate high-level scientific questions into executable workflows to acquire observations, prepare da...
VeriHarness is a method that enhances verification for large language model agents tackling long‑horizon tasks without needing reference answers at test time. It transforms the base LLM into an agentic verifier by providing a workspace, evidence tools, and reusable verification skills, using disagreement resolution and consensus challenge to evaluate competing claims. Across five benchmarks and two frontier models, VeriHarness outperforms baselines, achieving significant performance gains and demonstrating self‑improvement of verification skills from failure feedback.
The paper proposes a framework for agent calibration that ensures consistent capabilities when agents are deployed, migrated, or scaled across different models, harnesses, infrastructures, and user contexts. It defines standards for basic capabilities, technical environments, and user contexts, diagnoses gaps, applies revisions, and rechecks within fixed budgets, using diagnostic source behavior and mandatory tests to qualify agents. The framework includes semantic checkpoints, repair localization, and reusable configuration artifacts, and outlines an automatic calibration tool that relies on limited authorized user trajectories and tests, though empirical validation remains pending.
arXiv:2607. 23983v1 Announce Type: cross Abstract: Operational flood forecasting depends on tacit forecaster expertise that is difficult to formalize, audit, and transfer.
arXiv:2606. 31422v2 Announce Type: replace Abstract: Language agents acting over long horizons must maintain beliefs about tool states, object locations, graph edges, and subgoal dependencies.
arXiv:2607. 00436v1 Announce Type: new Abstract: Large language model agents are increasingly connected to scientific software, yet it remains unclear when tool access makes scientific computation more reliable rather than merely more complex.
arXiv:2607. 15781v1 Announce Type: new Abstract: Geospatial datasets support applications from urban planning to climate modeling, yet consistent assessment of FAIR compliance is difficult.
The article discusses how AI foundation models trained on weather and climate data are increasingly fine‑tuned for a wide range of Earth science tasks beyond traditional weather forecasting. It highlights that current evaluations rely mainly on benchmark skill metrics, which assess forecast accuracy but not the models’ representation of underlying physical processes, a gap that becomes critical under changing climate conditions. The authors propose five priorities for physical evaluation—covering training data, fine‑tuning, behavioral testing, mechanistic interpretability, and output validation—and recommend three actions for the next decade: open AI‑ready evaluation datasets, a shared physics‑based reporting standard, and a dedicated research program on model safety.
arXiv:2610.32441v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activitie...