arXiv AI
When a Kindergartener Solves Calculus: Measuring Capability Leakage in Role-Prompted Reasoning Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 10
Ask-E: An Environment for Calibrated Question Generation
arXiv:2608. 06933v1 Announce Type: cross Abstract: Today, we improve models by training and evaluating them on problems at the frontier of their abilities.
arXiv AI
Aug 3
The Self-Correction Illusion: Role Relabeling Gates Explicit Error Flagging in Large Language Models
arXiv:2606. 05976v2 Announce Type: replace Abstract: Recent works show that LLM agents struggle to correct errors in their own reasoning traces, despite their ability to correct errors from external sources.
arXiv AI
Jun 30
Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.
arXiv AI
Jun 6
Framing, Judging, Steering: An Assessable Competency Model for Teach-ing Students to Reason With Generative AI
arXiv:2606. 05983v1 Announce Type: new Abstract: Generative AI makes answers easy and understanding hard, and uncritical use invites cognitive offloading.
arXiv AI
Aug 25
GIM: Evaluating models via tasks that integrate multiple cognitive domains
The paper introduces the Grounded Integration Measure (GIM), a benchmark of 820 expert‑authored problems designed to test models on tasks that integrate multiple cognitive operations such as constraint satisfaction, state tracking, epistemic vigilance, and audience calibration. GIM emphasizes realistic, broadly accessible knowledge rather than specialized expertise, and uses a judge‑aware 2‑parameter logistic IRT model to produce robust ability estimates across 53 model‑thinking‑level configurations. The authors provide a comprehensive leaderboard of 22 models and 47 test configurations, and conduct an extensive study on how test‑time compute affects model capability, finding that configuration choices like thinking budget and quantization can be as influential as model selection itself. whyItMatters":"By focusing on integration of multiple cognitive domains, GIM offers a more realistic assessment of model reasoning capabilities than benchmarks that either overemphasize memorization or abstract reasoning alone."
arXiv Machine Learning
Sep 22
Characterizing Model-Native Skills
arXiv:2604.17614v2 Announce Type: replace-cross Abstract: Skills are a natural unit for describing what a language model can do and how its behavior can be changed. However, existing characterization...