The paper introduces a new evaluation for large language model agents called epistemic humility (EH), which measures an agent’s ability to recognize, act on, and communicate uncertainty when evidence conflicts with prior beliefs. EH is operationalized through three trajectory-level behaviors—Identify, Solve, and Escalate (ISE)—and tested in both controlled and naturally occurring knowledge conflict scenarios. Results show that higher task accuracy does not guarantee greater epistemic humility, and that model-level interventions can improve EH but often reduce accuracy, indicating that humility depends on the interplay between the backbone model, agent harness, and environment.
By Kaiser Sun, Bernal Jimenez Gutierrez, Hongjun Liu, Jingyu Zhang, Jie Gao, Mark Dredze, Daniel Khashabi
The paper introduces SCHEMA, a topology‑aware evaluation framework designed to assess hallucinations in large language model agents used for scientific research. SCHEMA builds scientific concept graphs from benchmark seeds and literature evidence, then generates graph‑grounded tasks such as claim verification, multi‑hop reasoning, open‑ended explanation, and experimental code generation. It evaluates agents using a trajectory hallucination pipeline that assigns a topology‑weighted severity score and a multi‑agent counterfactual attribution module that identifies the causal mechanisms behind failures. The study finds that hallucinations tend to cluster around highly connected knowledge hubs and that a model’s final accuracy can be misleading, as correct answers may arise from structurally flawed reasoning.
"whyItMatters":"The findings highlight that terminal accuracy alone is insufficient for high‑stakes scientific applications, underscoring the need for mechanism‑level evaluation grounded in knowledge topology."
By Xinshun Feng, Ziqi Miao, Lijun Li, Jing Shao
arXiv:2603. 09986v3 Announce Type: replace-cross Abstract: Hallucinations, the tendency for large language models to provide responses with factually incorrect and unsupported claims, is a serious problem within natural language processing for which we do not yet have an effective solution to mitigate against.
By Brandon C. Colelough, Davis Bartels, Dina Demner-Fushman
arXiv:2606. 26502v1 Announce Type: new Abstract: Large reasoning models (LRMs) take longer on harder problems, just as humans do.
By Han-yu Wang
arXiv:2606. 05976v1 Announce Type: new Abstract: Recent work shows that LLM agents struggle to correct errors in their own reasoning traces yet show markedly higher correction rates when identical claims appear under external sources.
By Kuan-Yen Chen, Fang-Yi Su, Jung-Hsien Chiang
Two prompts can request the same code change and produce the same correct patch, yet cause a coding agent to perform radically different kinds and amounts of work. We study this effect in a preregistered benchmark spanning 4,644 valid runs, 24 deterministic coding tasks, seven reasoning models, and two real agent harnesses.
The paper investigates how clinical large language models, when deployed as multi‑agent teams, fail to eliminate demographic bias through deliberation. Across five model families and four aggregation topologies on 120 MIMIC‑IV emergency cases, every configuration amplified or relocated bias, giving Black patients more acute triage. The authors term this phenomenon "disparity laundering" and show that only by making counterfactual demographic versions visible and pooling diverse perspectives at every deliberation round can the bias be substantially reduced without retraining or architectural changes.
By Xiaoyang Wang, Tianrui Wang, Christopher C. Yang
PHRBench is a new benchmark that evaluates how large language models handle hallucinated information during multi-stage reasoning. It tests 18 models across four domains using 4,820 controlled instances, measuring Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction to determine if a model can correct hallucinated premises and still reach the correct answer. The study finds that successful recovery is rare, often requires frequent belief updates, and that features of the hallucinated prompt can predict recovery with an AUROC of 0.847.
By Linghao Meng, Feng He, Xuan Yang, Junyuan Mao, Pinze Ren, Deqing Mu, Hesen Yang, Qiankun Li
arXiv:2601.22984v3 Announce Type: replace
Abstract: Diagnosing failure patterns in Deep Research Agents (DRAs) remains a critical challenge. Existing benchmarks predominantly rely on end-to-end evalu...
By Yuhao Zhan, Tianyu Fan, Linxuan Huang, Zirui Guo, Chao Huang
arXiv:2608. 14588v1 Announce Type: new Abstract: Sequential multi-agent LLM pipelines chain specialized agents without verification at handoffs, creating a structural flaw with measurable and severe consequences.
By Prabhjot Singh, Bhushan Pawar
Large reasoning models (LRMs) take longer on harder problems, just as humans do. This surface similarity hides an opposite pattern within items.
arXiv:2606. 19111v1 Announce Type: cross Abstract: Team science holds that leadership is contingent: it helps only under specific conditions, and capable, autonomous teams may need none at all.
By Haewoon Kwak