The paper introduces an information‑theoretic framework for evaluating knowledge tracing (KT) models that moves beyond aggregate metrics like AUC and accuracy. By applying Context Tree Weighting to item‑response histories, the authors create a causal uncertainty coordinate that separates observable uncertainty from unobserved local irreducible uncertainty (LIU). This approach allows model performance to be assessed across different entropy bands, revealing that modern KT models improve mainly in high‑entropy regions while also highlighting areas where apparent gains may be due to noise.
arXiv:2606. 14123v1 Announce Type: cross Abstract: Deployed knowledge-tracing models are typically frozen after training, yet systematic per-item logit bias arises, from limited per-item expressivity in backbone architectures and from post-deployment shifts in item properties, degrading prediction quality.
LUMOS is a diagnostic framework that tracks how knowledge in large language models (LLMs) originates from training data and manifests in outputs, using the fully transparent OLMo 2 corpus. The study finds that while models encode rare facts with high separability (84%), they often fail to express them behaviorally (54%), and self‑reflection accuracy drops sharply on unseen content. These results show that incorporating the training‑data axis into evaluation turns speculative claims into verifiable evidence, suggesting it should become a standard part of LLM knowledge assessment.
arXiv:2607. 27023v1 Announce Type: new Abstract: Evaluating large generative models across benchmarks is time-consuming and computationally expensive.
arXiv:2509. 21514v4 Announce Type: replace Abstract: Research on Knowledge Tracing (KT) models traditionally focuses on improving predictive accuracy.
The paper investigates how language models acquire knowledge over the course of pretraining by using point-in-time models trained on web snapshots from different years. It introduces a temporal variability score (TVS) that measures how consistently a knowledge item is represented across these snapshots. The study finds that items with high TVS improve more rapidly later in training and outperform low‑TVS items in most benchmark settings, especially on knowledge‑intensive tasks.
arXiv:2605.30018v3 Announce Type: replace-cross Abstract: Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of the...
The study evaluates 12 instruction‑tuned open‑weight LLMs on six causal‑graph benchmarks, testing five prompting strategies and four confidence sources. Findings show that LLMs tend to over‑predict edges, misclassify indirect or reversed edges as direct, and exhibit high over‑confidence, while conventional confidence estimates are unreliable and agreement signals offer limited improvement. The results suggest LLMs should be used as externally validated soft causal priors rather than definitive causal‑structure evidence.
arXiv:2607. 25257v1 Announce Type: cross Abstract: Item Response Theory (IRT) has recently been proposed as a framework for evaluating large language model (LLM) benchmarks by separating a model's latent ability from the properties of individual benchmark items.
arXiv:2610.17503v1 Announce Type: cross Abstract: Event logs are ubiquitous across real-world domains, including business processes, manufacturing workflows, healthcare, and many other operational sy...
arXiv:2607. 15190v1 Announce Type: new Abstract: AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate model capabilities, rank systems, select informative examples, and diagnose benchmark quality.
arXiv:2610.28522v1 Announce Type: new Abstract: Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make...
Newly developed items must ordinarily be field tested before their psychometric properties are known, creating a cold start problem for item calibration. Predicting item parameters from features is a long standing measurement problem dating back to the Linear Logistic Test Model; modern text embeddings now automate the design matrices traditionally specified by hand.