arXiv AI
Aug 10
Ask-E: An Environment for Calibrated Question Generation
arXiv:2608. 06933v1 Announce Type: cross Abstract: Today, we improve models by training and evaluating them on problems at the frontier of their abilities.
The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
The Token Games (TTG) is an evaluation framework that pits language models against each other by having them create and solve programming puzzles. Using pairwise duels and Elo ratings, the authors benchmarked ten frontier models, achieving rankings comparable to established tests while spending under $200 and requiring no human puzzle creation. The study highlights that generating high-quality puzzles remains difficult for current models and proposes a new paradigm for reasoning evaluation that also tests creativity and task creation.
Related stories
arXiv AI
Jun 16
Recurrent Reasoning on Symbolic Puzzles with Sequence Models
arXiv:2606. 15686v1 Announce Type: new Abstract: Large language models often appear strong on symbolic and algorithmic tasks, yet this apparent strength can hide brittle behaviour when problems become longer, harder, or slightly out of distribution.
arXiv AI
Oct 15
Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving
arXiv:2610.16133v1 Announce Type: cross Abstract: Puzzles offer a natural testbed for Large Language Models (LLMs) reasoning because they pose verifiable goals under partial information and require m...
arXiv AI
Oct 7
PuzzleJAX: A Benchmark for Reasoning and Learning
arXiv:2508.16821v2 Announce Type: replace Abstract: We introduce PuzzleJAX, a GPU-accelerated puzzle game engine and description language designed to support rapid benchmarking of tree search, reinfo...
arXiv AI
Aug 17
The Metacognitive Bottleneck: Japanese Riddles Reveal Fundamental Limits of Machine Insight and Self-Evaluation in Reasoning AI
arXiv:2509. 14704v3 Announce Type: replace Abstract: Benchmark saturation and training-data contamination increasingly obscure whether reported gains in large language models (LLMs) reflect genuine advances in reasoning or familiarity with recurring patterns in benchmark problems.
arXiv AI
Oct 1
ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning
arXiv:2609.38409v1 Announce Type: new Abstract: Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable r...
arXiv AI
Sep 17
WordPolo: Evaluating Language Models Through Iterative Semantic Feedback
WordPolo is a word‑finding task that evaluates language models by having them guess an unknown target word and receive semantic similarity feedback. Participants start with no knowledge, make iterative guesses, and receive distance scores that guide them through semantic space. The study tests recent LLMs, LRMs, humans, and a heuristic on 1,500 puzzles, revealing that while solve rates vary widely, many models make meaningful progress and exhibit human‑like strategies, highlighting the importance of assessing reasoning processes, not just final accuracy.
arXiv AI
Oct 2
Benchmarking Prompt Optimization of Large Language Models With Chess
arXiv:2610. 00416v1 Announce Type: new Abstract: Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamination, and assessing harder tasks can require expensive grading or execution infrastructure.
arXiv AI
6d ago
Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs
The paper investigates whether large language models (LLMs) can reliably grade Olympiad-level mathematical proofs. It introduces a human-curated dataset of USAMO 2025 candidate solutions with expert grading and evaluates 20 LLM verifiers across multiple metrics, prompt styles, and inference methods, finding that current models still have significant shortcomings. To improve performance, the authors propose TROJAN, an iterative self-critique pipeline for generating adversarial proofs, and MABGrader, a Multi-Armed Bandit framework that outperforms the state-of-the-art ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%.
arXiv AI
Oct 7
Universe of Thoughts: A Computational Framework for Creative Reasoning in Large Language Models
arXiv:2511.20471v3 Announce Type: replace Abstract: Recent advances in Large Language Model (LLM) reasoning have improved conventional problem solving, but creative reasoning remains comparatively un...
arXiv AI
Jun 24
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
arXiv:2501. 11790v5 Announce Type: replace-cross Abstract: Recent studies have raised significant concerns regarding the reliability of current mathematics benchmarks, highlighting issues such as simplistic design and potential data contamination.
arXiv AI
Jun 10
ComBench: A Benchmark for Rigorous Proof Reasoning and Constructive Realization in Olympiad-Level Combinatorics
arXiv:2606. 10479v1 Announce Type: new Abstract: Combinatorics is central to Olympiad-level mathematical problem solving, requiring deep discrete reasoning, creative constructions, and rigorous structural insight.