arXiv AI
Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 15
The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
The Token Games (TTG) is an evaluation framework that pits language models against each other by having them create and solve programming puzzles. Using pairwise duels and Elo ratings, the authors benchmarked ten frontier models, achieving rankings comparable to established tests while spending under $200 and requiring no human puzzle creation. The study highlights that generating high-quality puzzles remains difficult for current models and proposes a new paradigm for reasoning evaluation that also tests creativity and task creation.
arXiv AI
Jun 30
Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.
arXiv Computer Vision
Sep 15
PuzzleMate: Benchmarking MLLMs for Egocentric Puzzle Assistance
arXiv:2609.14473v1 Announce Type: new Abstract: Personal AI assistants hold the potential to evolve from digital interfaces into embodied companions capable of guiding users through complex physical...
arXiv AI
Aug 11
CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledge
arXiv:2604. 03374v2 Announce Type: replace-cross Abstract: Creative problem-solving requires combining multiple cognitive abilities, including logical reasoning, lateral thinking, analogy-making, and commonsense knowledge, to discover insights that connect seemingly unrelated pieces of information.
arXiv AI
Sep 17
WordPolo: Evaluating Language Models Through Iterative Semantic Feedback
WordPolo is a word‑finding task that evaluates language models by having them guess an unknown target word and receive semantic similarity feedback. Participants start with no knowledge, make iterative guesses, and receive distance scores that guide them through semantic space. The study tests recent LLMs, LRMs, humans, and a heuristic on 1,500 puzzles, revealing that while solve rates vary widely, many models make meaningful progress and exhibit human‑like strategies, highlighting the importance of assessing reasoning processes, not just final accuracy.
arXiv AI
Jun 8
DyCon: Dynamic Reasoning Control via Evolving Difficulty Modeling
arXiv:2606. 07108v1 Announce Type: new Abstract: Recent advances in Large Reasoning Models (LRMs) demonstrate remarkable performance improvements by iteratively reflecting, exploring, and executing complex tasks, yet suffer from inefficiencies due to redundant reasoning, known as "overthinking".