arXiv AI
Aug 18
RamseyGadgets: A Graph Construction Dataset for LLMs
arXiv:2608. 14999v1 Announce Type: cross Abstract: Constructing special graphs is an important task within graph theory and computer science.
PlanarBench: Evaluating LLM Spatial Reasoning via Planar Graph Drawing
arXiv:2606. 02010v1 Announce Type: cross Abstract: PlanarBench tests whether LLMs can draw planar graphs as ASCII art given only an edge list -- a spatial reasoning task that resists memorization because edge order, edge orientation, and node labels are all permutable.
Related stories
arXiv AI
6d ago
MathVis: Can AI Reason through Drawing? A Benchmark for Mathematical Figure Generation
MathVis is a new benchmark that tests large language models (LLMs) on their ability to generate and reason with mathematical figures. It contains 512 tasks, each with a mathematical statement, a reference diagram, executable TikZ or SVG code, a gold intermediate representation, atomic rubrics, and error codes. Initial results show that GPT‑6‑Astra, Claude Fable 5, and Gemini 3.1 Pro Preview achieve pass@1 scores of 31.25 %, 26.56 %, and 22.85 % respectively, yet the models tend to produce outputs dominated by formal languages and graphics programs rather than clear, diagram‑mediated reasoning, often resulting in visually confusing solutions for humans.
arXiv AI
Jun 2
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
arXiv:2602. 16902v4 Announce Type: replace Abstract: We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs).
arXiv AI
Jun 10
Lost in Serialization: Invariance and Generalization of LLM Graph Reasoners
arXiv:2511. 10234v3 Announce Type: replace-cross Abstract: While promising, graph reasoners based on Large Language Models (LLMs) lack built-in invariance to symmetries in graph representations.
arXiv AI
Aug 20
Solving Is Not Drawing: A Benchmark for Diagrammatic Reasoning in Olympiad Geometry
The paper introduces a new benchmark for diagrammatic reasoning in olympiad geometry, comprising 954 self‑contained problems and a 297‑problem hard subset. Each problem is paired with a human‑authored, high‑fidelity diagram in Asymptote code and a suite of metrics for evaluating diagram construction. Experiments show that current foundation models excel at solving the problems but produce markedly less faithful diagrams, with an average compile success rate of only 36.14%.
arXiv AI
Sep 10
Localizing and Correcting Errors for LLM-based Planners
arXiv:2602.00276v3 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated strong reasoning capabilities on math and coding, but frequently fail on symbolic classical planning...
arXiv Machine Learning
Jun 18
Formalizing and Mitigating Structural Distortion in LLM Attention for Graph Reasoning
arXiv:2606. 15633v2 Announce Type: replace Abstract: Large Language Models (LLMs) have shown promise for reasoning over Text-Attributed Graphs (TAGs).
Hugging Face Trending Papers
Jun 22
Graph-Enhanced Large Language Models for Spatial Search
There have been many recent improvements in the ability of Large Language Models (LLMs) to perform complex tasks and answer domain-specific questions through techniques like Retrieval Augmented Generation (RAG). However, reasoning abilities of LLMs, including spatial reasoning abilities, are still lacking.
arXiv Machine Learning
Jul 3
Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness
arXiv:2607. 01571v1 Announce Type: new Abstract: Chain-of-thought (CoT) reasoning enables large language models (LLMs) to solve complex problems by generating intermediate reasoning steps.
arXiv AI
Jul 28
Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planning
arXiv:2607. 22732v1 Announce Type: new Abstract: LLM-based game agents often perform poorly on more complex tasks.
Hugging Face Trending Papers
Oct 5
GraphDecide: Benchmarking System One Models on Graph Tasks
GraphDecide is a model‑independent benchmark designed to evaluate System One models—such as Jev—that make decisions directly from supplied options on graph‑related tasks. The benchmark combines structural task profiles, matched graph‑text input contrasts, and heuristic‑proposal controls to diagnose graph decision performance. In testing fourteen model‑interface configurations, GraphDecide shows that accurate adjacency recognition does not guarantee broader structural correctness, joint graph‑text input does not consistently improve prediction, and feasible construction does not ensure high solution quality.
arXiv AI
Jun 16
Beyond Scalars: Evaluating and Understanding LLM Reasoning via Geometric Progress and Stability
arXiv:2603. 10384v3 Announce Type: replace Abstract: Evaluating LLM reliability via scalar probabilities often fails to capture the structural dynamics of reasoning.