Do LLMs Understand Context? A Knowledge Graph-Based Evaluation Framework

The paper introduces a knowledge‑graph‑based evaluation framework, S3KG, to assess whether large language models truly understand context in question answering tasks. S3KG combines structural and semantic signals into a single similarity score and is paired with a diagnostic analysis that pinpoints reasoning errors at the triplet level. Across nine benchmarks, the method outperforms existing baselines, achieving up to +7.6 F1 points and an AUROC of 0.973.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• nlp
• benchmarks