← Back to all news
arXiv Machine Learning By Houru Jiang, Zixi Wang, Tengteng Cheng, Xueyi Li, Mingliang Hou, Jiaqi Zheng, Renqiang Luo, Teng Guo, Zitao Liu

An Information-Theoretic Evaluation Framework for Benchmark and Model Diagnosis in Knowledge Tracing

Read the original on arXiv Machine Learning →

The paper introduces an information‑theoretic framework for evaluating knowledge tracing (KT) models that moves beyond aggregate metrics like AUC and accuracy. By applying Context Tree Weighting to item‑response histories, the authors create a causal uncertainty coordinate that separates observable uncertainty from unobserved local irreducible uncertainty (LIU). This approach allows model performance to be assessed across different entropy bands, revealing that modern KT models improve mainly in high‑entropy regions while also highlighting areas where apparent gains may be due to noise.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jun 15

Recovering Stranded Discrimination in Knowledge Tracing: Per-Item Bias Correction via Empirical-Bayes Shrinkage

arXiv:2606. 14123v1 Announce Type: cross Abstract: Deployed knowledge-tracing models are typically frozen after training, yet systematic per-item logit bias arises, from limited per-item expressivity in backbone architectures and from post-deployment shifts in item properties, degrading prediction quality.

By Xiaoran Yan, Cheng Tang, Atsushi Shimada
safety
More like this →
arXiv AI
Oct 5

LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs

LUMOS is a diagnostic framework that tracks how knowledge in large language models (LLMs) originates from training data and manifests in outputs, using the fully transparent OLMo 2 corpus. The study finds that while models encode rare facts with high separability (84%), they often fail to express them behaviorally (54%), and self‑reflection accuracy drops sharply on unseen content. These results show that incorporating the training‑data axis into evaluation turns speculative claims into verifiable evidence, suggesting it should become a standard part of LLM knowledge assessment.

By Seoyeon Ye, Gayoung Kim, Jiyoung Hong, Sookyung Kim, Hyunsoo Cho
llms
More like this →
arXiv AI
Oct 28

What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

The paper investigates how language models acquire knowledge over the course of pretraining by using point-in-time models trained on web snapshots from different years. It introduces a temporal variability score (TVS) that measures how consistently a knowledge item is represented across these snapshots. The study finds that items with high TVS improve more rapidly later in training and outperform low‑TVS items in most benchmark settings, especially on knowledge‑intensive tasks.

By Chang Su, Bin Wu, Emine Yilmaz, Yao Lu
llmsbenchmarks
More like this →
arXiv Machine Learning
Oct 8

Latent Performance Profiling of Large Language Models

arXiv:2605.30018v3 Announce Type: replace-cross Abstract: Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of the...

By Tanmoy Chakraborty, Ayan Sengupta, Suparna Bhattacharya, Partha Pratim Chakrabarti, Amlan Chakrabarti, Supratik Chakraborty, Partha Pratim Das, Lipika Dey, Richa Singh, Mayank Vatsa
llmsbenchmarkssafety
More like this →