GraphNIAH is a new benchmark for long‑context reasoning that uses real citation relationships among scientific papers published after a model’s knowledge cutoff. It automatically samples subgraphs of the citation graph and turns each into a long‑context QA example with a verifiable answer, avoiding synthetic data, dataset reuse, manual annotation, and trivial reasoning steps. Evaluation of eight large language models shows that accuracy drops sharply as reasoning depth increases, indicating that current models often fail to fully explore citation relationships even when all necessary facts are present.
arXiv:2509. 21028v4 Announce Type: replace Abstract: We introduce SciTrek, a synthetic question-answering dataset for assessing and improving long-context numerical reasoning in large language models (LLMs).
HyGRAIL is a framework for discovering scientific hypotheses in incomplete knowledge graphs by combining a graph neural network (GNN) triage with large language model (LLM) review. The GNN scores candidate hypotheses and routes only ambiguous cases to the LLM, which receives structured evidence from the graph converted into natural language. Experiments on MatKG show HyGRAIL achieves the highest F1 score, improves over baselines, and cuts LLM calls by over 54%.
arXiv:2607. 08284v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them.
Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness.
arXiv:2608. 18242v1 Announce Type: new Abstract: We introduce ClosureBench, a constructive benchmark for compositional graph-relational reasoning with programmatically verified ground truth.
arXiv:2602.02518v3 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly rely on external knowledge to improve factuality, yet many knowledge sources on the Web, such as sc...
Scientific knowledge graphs organize entities and relations extracted from scientific literature, but they remain inherently incomplete. Missing typed links in such graphs can therefore represent plau...
LongForge introduces a systematic way to evaluate and generate long‑context instruction‑tuning data by constructing evidence‑assertion reasoning graphs for each sample. The analyzer measures four dimensions—breadth, depth, correctness, and completeness—making the data auditable and revealing that breadth does not increase with document length while depth remains low. The synthesizer uses these metrics to control the training distribution, and two variants, LongForge‑1 and LongForge‑N, outperform the best baseline by 2.2–4.7 points across six benchmarks, with an additional 1.16‑point gain when explicitly increasing breadth and depth.
arXiv:2609.22939v1 Announce Type: cross Abstract: Long-context models read a novel the way a person reads a printout: one token after another, in narrative order, with the whole history competing for...
arXiv:2603.00889v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, largely enabled by supervised fine-tuning (SFT)- and reinforce...
arXiv:2608. 03292v1 Announce Type: new Abstract: Long Document Visual Question Answering (LongDocVQA) requires Multimodal Large Language Models (MLLMs) to locate, integrate, and reason over heterogeneous document elements distributed across multiple pages.
LiveMathematicianBench is a dynamic multiple‑choice benchmark for research‑level mathematical reasoning, built from recent arXiv papers published after model training cutoffs. It introduces a thirteen‑category logical taxonomy of theorem types and uses a proof‑sketch‑guided distractor pipeline to create plausible but invalid answer choices, enhancing sensitivity to genuine reasoning. Evaluation shows current large language models perform poorly, with the best model scoring 43.5% overall and only 17.6% under substitution‑resistant conditions, indicating the benchmark’s difficulty and realism.