← Back to all news
arXiv AI By Tal Zeevi, Trey W. Jensen, Maxwell Strome

When Scientific Contradictions Are Lost in Translation

Read the original on arXiv AI →

The study examines how language models decide whether two scientific findings conflict by translating an unsatisfiable XOR constraint system into reports from different laboratories. When constraints are explicit, GPT‑5.6 Sol and Claude Opus 5 correctly select the best-supported assignment in 90% and 96% of cases, respectively. In scientific prose, Claude Opus 5 tends to favor biologically expected assignments, but this bias can be mitigated by formalization requests and explicit paired‑design cues, raising correct recovery to 92%.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 30

Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy

The paper introduces a framework for evaluating large language model agents by attempting to end‑to‑end reproduce published astronomy studies, separating execution from verification and distinguishing computational failures from methodological ambiguities. Applying this to fourteen papers—one from The Astrophysical Journal and thirteen from Nature—revealed that eleven contained ambiguities that prevented a uniquely specified reproduction path. In a controlled case study, twelve different analysis paths produced distance estimates ranging from 2.16 to 3.53 kpc, with only one matching the published value of ~2.70 kpc, demonstrating that matching outcomes does not guarantee that the agent has reconstructed the underlying reasoning. whyItMatters":"The study shows that end‑to‑end reproduction can expose gaps in implicit scientific knowledge within AI systems, highlighting the need for better integration of causal relevance in LLM agents."

By Yuehui Wang, Xinyu Qi, Guirong Xue, Cheng Wang, Yangbin Xie, Xiaoyu Tang, Cong Sun
llmsagents
More like this →
arXiv AI
Oct 7

Learning to Outgrow a Theory: Experimental Discovery Beyond the Initial Hypothesis Space

The paper introduces an experimental model‑class revision framework that jointly proposes structural edits to a hypothesis space and diagnostic experiments to test those edits. By coupling a class‑level distinguishability objective with anytime‑valid sequential evidence, the method only revises the model class after the current one is rejected. On 400 controlled dynamical environments, the approach achieves 89.5% exact recovery with 32 experiments, outperforming baselines and transferring well to unseen mechanisms, library insufficiency detection, and other benchmark tasks.

By SiYuan Ma, Albert Gao, Chunzheng Zhu, Xin Yan, Wenlong Zhang, Wenxin Zhang, Luqi Gong, Tianlin Li, Qixin Zhang
More like this →
arXiv AI
Aug 3

ModelEquivBench: Certifying Multi-Relational Evaluation of LLM-Generated Optimization Models

arXiv:2607. 29431v1 Announce Type: new Abstract: Large language models increasingly generate optimization models from natural language, but existing evaluation often reduces a generated model and its ground truth to a single equivalent/not-equivalent verdict or an execution-success rate--labels that are neither independently checkable nor faithful to the multiple distinct senses in which two formulations can agree.

By Penglin Zhu, Jungang Xu
llmssafety
More like this →