GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs

GraphNIAH is a new benchmark for long‑context reasoning that uses real citation relationships among scientific papers published after a model’s knowledge cutoff. It automatically samples subgraphs of the citation graph and turns each into a long‑context QA example with a verifiable answer, avoiding synthetic data, dataset reuse, manual annotation, and trivial reasoning steps. Evaluation of eight large language models shows that accuracy drops sharply as reasoning depth increases, indicating that current models often fail to fully explore citation relationships even when all necessary facts are present.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• benchmarks