arXiv AI

OA-RAG: Ontology-based Agentic RAG for Faithful LLM Reasoning

arXiv AI

Oct 2

Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI

The paper introduces a pipeline that automatically creates ontology‑grounded multiple‑choice question benchmarks for evaluating large language models (LLMs) on logical reasoning tasks in scientific AI. By using OWL 2 ontologies, correct answers are guaranteed by design and distractors are generated and formally verified as incorrect through an OWL reasoner. Experiments on three ontologies—Pizza, PMDco, and DOID—yielded 112, 2,491, and 15,216 MCQs, respectively, with high natural‑language quality and challenging zero‑shot performance for six LLMs.

By Nishtha N. Vaidya, Stephan Grimm, Thomas Hubauer, Thomas A. Runkler

llms rag nlp benchmarks

arXiv AI

Aug 25

Toward Effective and Reliable LLM Agents via Dynamic Ontology

arXiv:2608.22974v1 Announce Type: new Abstract: Large language model (LLM) agents rely heavily on knowledge encoded in model parameters or presented as unstructured context. In domain-specific tasks,...

By Xiaohui Zhang, Zequn Sun, Chengyuan Yang, Yuanning Cui, Lingbing Guo, Wei Hu

llms agents

arXiv AI

Jul 21

Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generation

arXiv:2607. 18029v1 Announce Type: cross Abstract: Researchers need to answer ad-hoc questions about the contents of domain-specific archives but often lack the expertise to write structured queries on the metadata.

By Blake G. Fitch, Cato Elia Kurtz

llms agents fine-tuning benchmarks safety data-engineering

arXiv AI

Jun 9

From Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAG

arXiv:2603. 03292v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) exhibit high reasoning capacity in medical question-answering, but their tendency to produce hallucinations and outdated knowledge poses critical risks in healthcare fields.

By Wenhao Wu, Zhentao Tang, Yafu Li, Shixiong Kai, Mingxuan Yuan, Zhenhong Sun, Chunlin Chen, Zhi Wang

llms rag agents reinforcement-learning benchmarks safety

arXiv AI

Jul 28

Evo-DKD: Dual-Knowledge Decoding for Autonomous Ontology Evolution in Large Language Models

arXiv:2507. 21438v2 Announce Type: replace Abstract: Ontologies and knowledge graphs require continuous evolution to remain comprehensive and accurate, but manual curation is labor intensive.

By Vishal Raman, Vijai Aravindh R, Abhijith Ragav

llms rag agents

arXiv AI

Oct 7

MedZERO: Self-Evolving Agents for Open-Ended Medical Reasoning Through Controlled Knowledge Accumulation

arXiv:2610.08327v1 Announce Type: new Abstract: Large language models (LLMs) have shown promise in medical question answering and clinical reasoning, yet their improvement remains constrained by stat...

By Xilin Dang, Weilin Ruan, Xue Yang, Jinghao Wang, Xiaowei Hu, Jinpeng Li, Pheng-Ann Heng

llms agents nlp benchmarks

arXiv AI

Jul 23

Reference-Free Evaluation of Reasoning in Open-Ended Question Answering

arXiv:2607. 19678v1 Announce Type: cross Abstract: AI-generated answers in high-stakes domains are often fluent but difficult to verify, especially when they contain multi-step reasoning rather than a single final answer.

By Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull, Gregory E Dean, Maria Liakata

llms nlp benchmarks

arXiv AI

Oct 7

Rethinking Knowledge Retrieval for Generation: A Survey on RAG Architectures and Applications

arXiv:2610.01936v2 Announce Type: replace Abstract: Large Language Models (LLMs) have demonstrated remarkable fluency and versatility across natural language tasks but remain fundamentally limited by...

By Meghana Sunil, Shravya V, Shravan Venkatraman, Joe Dhanith PR

llms rag reinforcement-learning safety

arXiv AI

Jul 8

From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b

arXiv:2607. 06452v1 Announce Type: cross Abstract: Biomedical question answering requires not only accurate extraction of information from scientific literature but also reliable integration of evidence across multiple documents.

By Taeyun Roh, Eunha Lee, Wonjune Jang, Sohyun Chung, Junha Jung, Jaewoo Kang

llms agents nlp

arXiv AI

Aug 28

Do Language Models Follow Occam's Razor? An Evaluation of Parsimony in Inductive and Abductive Reasoning

The paper investigates whether large language models (LLMs) follow Occam's Razor when performing inductive and abductive reasoning. It introduces a synthetic framework for generating questions that require both types of reasoning and a new automated metric to evaluate the simplicity and correctness of generated hypotheses. Experiments show that while LLMs can handle simple scenarios, they struggle with complex world models and producing high‑quality, simplest hypotheses, even when using advanced reasoning techniques.

By Yunxin Sun, Abulhair Saparov

llms agents benchmarks

arXiv Computation and Language

Sep 23

Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models

arXiv:2609.26610v1 Announce Type: new Abstract: Despite their outstanding performance on many NLP tasks, LLMs face serious challenges related to semantic abstraction. In this study, we are interested...

By David Torres-Moreno, Jorge Hermosillo-Valadez

llms agents nlp

arXiv AI

Jun 2

KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models

arXiv:2604. 17621v2 Announce Type: replace Abstract: Many real-world questions appear deceptively simple yet implicitly demand two capabilities: (i) systematic coverage of a bounded knowledge universe and (ii) compositional set-based reasoning over that universe, a phenomenon we term "the tip of the iceberg.

By Xiao Zhang, Qianru Meng, Yongjian Chen, Yumeng Wang, Johan Bos

llms benchmarks

Related stories