arXiv AI

PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks

PotARCin expands the ARC benchmark by evaluating abstract reasoning across five dimensions—Definition, Classification, Constrained Generation, Editing, and Inversion—using programmatic generation of new task instances. The study shows a 25‑52 percentage‑point performance gap between standard ARC evaluation and PotARCin, and reveals that multi‑dimensional assessment can reorder models that appear equivalent under single‑metric accuracy. Additionally, a new held‑out set, P‑ARC, demonstrates low model accuracy (1‑8%) across all dimensions, highlighting the need for more comprehensive tests of abstract reasoning.

arXiv AI

Sep 21

Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

The paper introduces a two‑step test‑time training protocol, Embed‑TTT, for Vision ARC (VARC) that first fine‑tunes only the task embedding and then fine‑tunes the backbone. This approach consistently produces task embeddings that better align with the underlying rules, improves retrieval and linear probing, and recovers the geometric structure of parametric rules. Even fine‑tuning only the tiny embedding component solves a significant portion of ARC‑AGI‑1, ConceptARC, and Mini‑ARC tasks, while the full two‑step pipeline further enhances performance and demonstrates compositional rule interpolation.

By Adrien Deli\`ege, Claas Beger, Marc Van Droogenbroeck, Melanie Mitchell

rag benchmarks

arXiv AI

Aug 17

The Metacognitive Bottleneck: Japanese Riddles Reveal Fundamental Limits of Machine Insight and Self-Evaluation in Reasoning AI

arXiv:2509. 14704v3 Announce Type: replace Abstract: Benchmark saturation and training-data contamination increasingly obscure whether reported gains in large language models (LLMs) reflect genuine advances in reasoning or familiarity with recurring patterns in benchmark problems.

By Masaharu Mizumoto, Dat Nguyen, Zhiheng Han, Xingfu Li, Yo Nakawake, Le Minh Nguyen

llms benchmarks

arXiv AI

Jun 30

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.

By Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai, Dengfeng Liu, Qiaobo Hao, Huang Liu, Zhiguo Yang, Ji Xie, Ninglun Gu, Jin Yang, Kailai Zhang, Yelun Bao, Jun Wang

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jul 8

Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges

arXiv:2605. 19723v2 Announce Type: replace-cross Abstract: Mathematical reasoning is essential for problem-solving in education, science, and industry, serving as a crucial benchmark for evaluating artificial intelligence systems.

By Husnain Amjad, Raja Khurram Shahzad, Aamir Shahzad, Mehwish Fatima

llms fine-tuning benchmarks

arXiv AI

Jun 29

DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models

arXiv:2606. 26530v2 Announce Type: replace-cross Abstract: The Abstraction and Reasoning Corpus (ARC) contains tasks that require summarizing patterns from limited grid samples and predicting output grids.

By Yuxuan Yang, Feiyang Li, Yile Wang

llms fine-tuning benchmarks safety

arXiv AI

Sep 15

Post-Reasoning: Improving the Performance of Non-Thinking Models at No Cost

arXiv:2605.06165v2 Announce Type: replace Abstract: As the widespread adoption of Large Language Models (LLMs) accelerates, token consumption from intermediate reasoning traces increasingly contribut...

By Richmond Sin Jing Xuan, Rishabh Bhardwaj, Soujanya Poria

llms efficiency benchmarks

arXiv AI

Jun 26

\textsc{DiARC}: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models

arXiv:2606. 26530v1 Announce Type: cross Abstract: The Abstraction and Reasoning Corpus (ARC;~\citealp{chollet2019measure}) contains tasks that require summarizing patterns from limited grid samples and predicting output grids.

By Yuxuan Yang, Feiyang Li, Yile Wang

llms fine-tuning benchmarks safety

arXiv AI

Jun 8

CORE: Contrastive Reflection Enables Rapid Improvements in Reasoning

arXiv:2605. 28742v2 Announce Type: replace Abstract: Language models can use verifiable rewards to improve at a wide variety of reasoning tasks.

By Linas Nasvytis, Simon Jerome Han, Ben Prystawski, Satchel Grant, Noah D. Goodman, Judith E. Fan

llms rag

arXiv Machine Learning

Aug 11

Rethinking Reasoning with MDLMs: Early Exits, Post-hoc Reasoning, and Beyond

arXiv:2510. 19990v2 Announce Type: replace Abstract: The reasoning paradigm, where language models reason before answering, has enabled breakthroughs on tasks such as mathematical problem-solving.

By Zachary Horvitz, Raghav Singhal, Hao Zou, Carles Domingo-Enrich, Zhou Yu, Rajesh Ranganath, Kathleen McKeown

llms diffusion reinforcement-learning fine-tuning

arXiv AI

Aug 12

Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers

arXiv:2604. 07822v2 Announce Type: replace-cross Abstract: We study implicit reasoning, i.

By Harsh Kohli, Srinivasan Parthasarathy, Huan Sun, Yuekun Yao

llms

arXiv AI

Jun 2

TInR: Exploring Tool-Internalized Reasoning in Large Language Models

arXiv:2604. 10788v2 Announce Type: replace-cross Abstract: Tool-Integrated Reasoning (TIR) has emerged as a promising direction by extending Large Language Models' (LLMs) capabilities with external tools during reasoning.

By Qiancheng Xu, Yongqi Li, Fan Liu, Hongru Wang, Min Yang, Wenjie Li

llms reinforcement-learning fine-tuning safety

arXiv AI

Jun 12

Constructing Evaluation Datasets for Procedural Reasoning: Balancing Naturalness, Grounding, and Multi-Hop Coverage

arXiv:2606. 12767v1 Announce Type: new Abstract: Evaluating procedural reasoning in AI-supported learning systems requires question-answer datasets that are both learner-like and grounded in the instructional knowledge the system is expected to use.

By Sarah Elshabrawy, Rahul K. Dass, Ashok K. Goel

Related stories