The study evaluates hybrid retrieval, NLI, and open‑source LLMs for scientific claim verification. Using a benchmark of 2,035 claims, the authors compare a DeBERTa‑based NLI baseline with Llama 3.1, Qwen3, and Mistral LLMs, finding that Qwen3 achieves the highest Macro‑F1 score. A conservative fusion rule that labels a claim as Support or Contradict only when both models agree actually degrades performance, and error analysis reveals that disagreements often overturn correct LLM predictions.
By Lakshmi Narayanan P, Narenkarthik K, Naren Kumar C
The paper investigates how large language models (LLMs) interpret negation across different logical semantics—open‑world vs. closed‑world, two‑ vs. three‑valued, and credulous vs. skeptical reasoning. Using the newly introduced NAFBench, a procedural generator that creates solver‑certified logic programs and their natural‑language verbalizations, the authors evaluate LLMs on four semantic viewpoints (SLDNF, well‑founded semantics, and stable‑model semantics). Results show a persistent gap: even the strongest models achieve only 59–74% accuracy, with many models sensitive to rule ordering and prone to overcommitment on undefined cases, though some frontier models reach near‑perfect performance on a fixed‑complexity set.
"whyItMatters":"The study highlights that current LLMs struggle to reliably follow explicitly specified negation semantics, underscoring a limitation in their logical reasoning capabilities."
By Qiming Bao, Agnieszka Mensfelt, Michael J. Witbrock, Kostas Stathis
arXiv:2606. 29034v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly summarize clinical evidence, where a claim's weight depends on how strongly it is supported.
By Soroosh Tayebi Arasteh
The study examines how different editorial framings in prompts influence large language models’ statistical analysis reports. Using a 4×4 factorial design, researchers found that certain framings—particularly brutally critical prompts on genuine effects and significance-seeking prompts on underpowered nulls—led to factual misrepresentations. Tone shifts were more widespread, with critical framing inducing defensive language across all data patterns, while a confound in the data largely prevented both factual and tonal distortions.
By Paras Balani, Subhrakanta Panda
The paper presents a clinician‑led safety guardrail for mental‑health chatbots that replaces typical prompt‑engineering or Constitutional AI methods. Two clinicians design, finalize, and adjudicate a rule base of four safety checks, which is formally verified for internal consistency and applied to LLM outputs. The verification layer’s verdicts are recomputable from a named rule and sixteen facts, and it consistently avoids false crisis‑resource demands that a matched LLM judge frequently raises.
By Chenhan Lyu, Kit Wislocki, Sarah Elizabeth Stoeckl, Maitreyi Sinha, Amir Rahmani
arXiv:2507. 09751v3 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but exhibit problems with logical consistency in their output.
By Bradley P. Allen, Prateek Chhikara, Thomas Macaulay Ferguson, Filip Ilievski, Paul Groth
The paper introduces a typed reasoning graph framework to compare human and large language model (LLM) reasoning paths in scientific fact‑checking. By modeling explanations as graphs linking false claims to study context, findings, premises, and fallacy labels, the authors enable one‑to‑one alignment of human and LLM reasoning at the sub‑graph level. Using 84 false claims from MISSCIPLUS, they evaluate GPT‑5, Claude Opus 4.7, and Qwen3‑32B, finding distinct performance patterns: Qwen3‑32B has the lowest verdict failure rate, GPT‑5 shows the highest human alignment, and Claude Opus 4.7, while weak at verdict prediction, often produces valid reasoning in successful cases.
By Abdul Ghafoor, Muhammad Arslan Manzoor, Yufang Hou
arXiv:2607.15870v2 Announce Type: replace
Abstract: Human label variation in natural language inference is increasingly treated as signal rather than noise, but how much of it formal semantic structu...
By Haram Choi (University of Bremen)
LogiMed‑RoB is a new benchmark that tests large language models (LLMs) on hierarchical logical consistency in medical risk‑of‑bias assessments, using 860 randomized controlled trials and 14,820 queries based on Cochrane Risk of Bias 2.0 expert logic. The benchmark evaluates models across four dimensions—Atomic Consistency, Domain Consistency, Aggregation Consistency, and Evidential Faithfulness—revealing a catastrophic error‑compounding effect where high atomic accuracy does not translate to end‑to‑end consistency. Experiments on ten state‑of‑the‑art LLMs show that even top models can fail to deduce correct outcomes in a significant portion of cases, highlighting a gap between evidence retrieval and reasoning.
whyItMatters":"The study shows that high outcome accuracy can mask critical reasoning flaws, emphasizing the need for white‑box logical verification before deploying LLMs in clinical settings."
By Jiayu Huang, Zichen Tang, Qianhui Ling, Zemin Kuang, Haihong E
arXiv:2607. 26102v1 Announce Type: cross Abstract: Mathematical chain of thought (CoT) evaluation is commonly reduced to whether the final answer matches a reference.
By Vivek Shukla, Varun Shukla, Atul, Divya Mishra, Mehul Kumar Das
The paper investigates why retrieval‑based open‑ended evaluation fails in medical fact verification. By creating two detailed taxonomies—one for retrieval‑stage errors across five quality dimensions and another for verifier‑reasoning errors across six steps—the authors automatically label evidence quality and reasoning errors using an LLM‑as‑Judge pipeline. Their large‑scale stress tests across multiple retrieval methods and verifier models show that increasing model size, reasoning effort, source breadth, or medical fine‑tuning does not eliminate these failure modes, indicating fundamental limits of the retrieve‑then‑verify paradigm in open‑ended medical contexts.
By Heyuan Huang, Jirui Dai, Alexandra DeLucia, Sonal Joshi, Mahsa Yarmohammadi, Jie Gao, Bernal Jim\'enez Guti\'errez, Mark Dredze
arXiv:2609.37788v1 Announce Type: cross
Abstract: Rubrics support the structured evaluation of language models. We propose a rubric for assessing expressed clinical reasoning in model responses, draw...
By Zhangshu Joshua Jiang, Zina Ibrahim, James T. Teo