The paper introduces a knowledge‑graph‑based evaluation framework, S3KG, to assess whether large language models truly understand context in question answering tasks. S3KG combines structural and semantic signals into a single similarity score and is paired with a diagnostic analysis that pinpoints reasoning errors at the triplet level. Across nine benchmarks, the method outperforms existing baselines, achieving up to +7.6 F1 points and an AUROC of 0.973.
By Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe, Chamath Gunapala, Pragatheeswaran Vipulanandan, Kamal Premaratne, Uthayasanker Thayasivam
arXiv:2606. 06197v1 Announce Type: cross Abstract: Question answering (QA) systems have achieved notable progress with the advent of large language models (LLMs).
By Hafez Abdelghaffar, Ahmed Alansary, Ali Hamdi
arXiv:2608. 07838v1 Announce Type: new Abstract: Large language models (LLMs) have increasingly supported response generation grounded in user-provided knowledge spanning heterogeneous structures.
By Shibo Chu, Yuze Liu, Tiehua Zhang, Zhishu Shen, Lianghua He, Haofen Wang, Zhijun Ding
arXiv:2606. 19351v1 Announce Type: cross Abstract: Knowledge graph (KG) reasoning infers new knowledge from existing facts and is widely applied in question answering, recommendation, and decision support.
By Xinyan Zhu, Yaoqi Liu, Yue Gao, Huadong Ma, Cheng Yang, Chuan Shi
arXiv:2510. 06039v2 Announce Type: replace-cross Abstract: Reliable evaluation of knowledge-grounded Large Language Models (LLMs) in Chinese requires resources that explicitly align Chinese-language text with verifiable Knowledge Graph (KG) facts.
By Chengwei Wu, Xingrui Zhuo, Mingyang Gao, Xinghe Cheng, Zhichao Yan, Jiapu Wang
The paper introduces KBevo, a co‑evolving framework that simultaneously builds a structured knowledge base and performs reasoning over it for knowledge‑intensive question answering. By optimizing both components end‑to‑end with QA outcome rewards, the system improves the quality and connectivity of the knowledge base, leading to higher answer reachability and better compositional factual reasoning. Compared to standard retrieval baselines, KBevo offers greater controllability and improved factual accuracy.
By Ryan Thomas Noonan, Linxi Zhao, Menghan Xu, Akanksha Sarkar, Mihir Mishra, Dongyoung Go, Kilian Q. Weinberger, Yoav Artzi, Jennifer J. Sun
arXiv:2506. 06331v2 Announce Type: replace-cross Abstract: By retrieving contexts from knowledge graphs, graph-based retrieval-augmented generation (GraphRAG) enhances large language models (LLMs) to generate quality answers for user questions.
By Qiming Zeng, Hao Luo, Yuhao Lin, Yicheng Jin, Yuxiang Wang, Fangcheng Fu, Xiao Yan, Jiawei Jiang
arXiv:2606. 14047v1 Announce Type: cross Abstract: Long-context language modeling requires not only extending context windows but maintaining coherent understanding of entity states and relationships across thousands of tokens -- a challenge that semantic similarity alone cannot address.
By Ghadir Alselwi, Basem Suleiman, Hao Xue, Shoaib Jameel, Hakim Hacid, Flora D. Salim, Imran Razzak
The paper introduces a trace-level divergence framework for evaluating multilingual large language models, breaking down agreement into structural similarity, order consistency, and holistic similarity. Experiments across three reasoning benchmarks, seven languages, and six leading LLMs show that conventional holistic similarity metrics overstate agreement, with up to 67.3% of top-scoring examples lacking structural alignment. Human judgments align more closely with structural similarity, highlighting language-specific divergence and the need for trace-level evaluation in multilingual deployments.
By Rui Sun, Peizhao Mei, Xiewei Cheng, Kaizhuo Chen, Taoran Li, Zhiyuan Yu, Varun Chandrasekaran
arXiv:2511.04473v3 Announce Type: replace
Abstract: Retrieval of information from graph-structured knowledge bases represents a promising direction for improving the factuality of LLMs. While various...
By Alberto Cattaneo, Carlo Luschi, Daniel Justus
arXiv:2608. 15535v1 Announce Type: cross Abstract: We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs).
By Rinit Jain, Tirthraj Mahajan, Advait Joshi, Raviraj Joshi
arXiv:2607. 06940v1 Announce Type: cross Abstract: The remarkable performance of large language models (LLMs) in linguistic tasks underscores an urgent need for comprehensive evaluation of their response quality.
By Yiming Gai, Junde Lu, Xuefei Huang