arXiv AI September 24, 2026 By Claas Beger, Ryan Yi, Melanie Mitchell

PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks

PotARCin expands the ARC benchmark by evaluating abstract reasoning across five dimensions—Definition, Classification, Constrained Generation, Editing, and Inversion—using programmatic generation of new task instances. The study shows a 25‑52 percentage‑point performance gap between standard ARC evaluation and PotARCin, and reveals that multi‑dimensional assessment can reorder models that appear equivalent under single‑metric accuracy. Additionally, a new held‑out set, P‑ARC, demonstrates low model accuracy (1‑8%) across all dimensions, highlighting the need for more comprehensive tests of abstract reasoning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

benchmarks

arXiv AI

Sep 21

Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

The paper introduces a two‑step test‑time training protocol, Embed‑TTT, for Vision ARC (VARC) that first fine‑tunes only the task embedding and then fine‑tunes the backbone. This approach consistently produces task embeddings that better align with the underlying rules, improves retrieval and linear probing, and recovers the geometric structure of parametric rules. Even fine‑tuning only the tiny embedding component solves a significant portion of ARC‑AGI‑1, ConceptARC, and Mini‑ARC tasks, while the full two‑step pipeline further enhances performance and demonstrates compositional rule interpolation.

By Adrien Deli\`ege, Claas Beger, Marc Van Droogenbroeck, Melanie Mitchell

rag benchmarks

arXiv AI

Aug 17

The Metacognitive Bottleneck: Japanese Riddles Reveal Fundamental Limits of Machine Insight and Self-Evaluation in Reasoning AI

arXiv:2509. 14704v3 Announce Type: replace Abstract: Benchmark saturation and training-data contamination increasingly obscure whether reported gains in large language models (LLMs) reflect genuine advances in reasoning or familiarity with recurring patterns in benchmark problems.

By Masaharu Mizumoto, Dat Nguyen, Zhiheng Han, Xingfu Li, Yo Nakawake, Le Minh Nguyen

llms benchmarks

arXiv AI

Jun 30

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.

By Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai, Dengfeng Liu, Qiaobo Hao, Huang Liu, Zhiguo Yang, Ji Xie, Ninglun Gu, Jin Yang, Kailai Zhang, Yelun Bao, Jun Wang

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jul 8

Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges

arXiv:2605. 19723v2 Announce Type: replace-cross Abstract: Mathematical reasoning is essential for problem-solving in education, science, and industry, serving as a crucial benchmark for evaluating artificial intelligence systems.

By Husnain Amjad, Raja Khurram Shahzad, Aamir Shahzad, Mehwish Fatima

llms fine-tuning benchmarks

arXiv AI

Jun 29

DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models

arXiv:2606. 26530v2 Announce Type: replace-cross Abstract: The Abstraction and Reasoning Corpus (ARC) contains tasks that require summarizing patterns from limited grid samples and predicting output grids.

By Yuxuan Yang, Feiyang Li, Yile Wang

llms fine-tuning benchmarks safety

arXiv AI

Sep 15

Post-Reasoning: Improving the Performance of Non-Thinking Models at No Cost

arXiv:2605.06165v2 Announce Type: replace Abstract: As the widespread adoption of Large Language Models (LLMs) accelerates, token consumption from intermediate reasoning traces increasingly contribut...

By Richmond Sin Jing Xuan, Rishabh Bhardwaj, Soujanya Poria

llms efficiency benchmarks

One email a morning, machine-written

Related stories

Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

The Metacognitive Bottleneck: Japanese Riddles Reveal Fundamental Limits of Machine Insight and Self-Evaluation in Reasoning AI

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges

DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models

Post-Reasoning: Improving the Performance of Non-Thinking Models at No Cost