arXiv:2610.14542v1 Announce Type: new Abstract: Multimodal benchmarks are central to evaluating whether models can reason over information from multiple modalities. However, existing literature has r...
Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or...
arXiv:2610.28522v1 Announce Type: new Abstract: Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make...
arXiv:2608.30372v1 Announce Type: new Abstract: As Large Language Models rapidly advance, performance on standard multiple-choice question answering (MCQA) benchmarks is reaching saturation. While th...
arXiv:2606. 00301v1 Announce Type: new Abstract: Hallucinations in large language models (LLMs) arise from heterogeneous failure mechanisms, making reliable detection difficult for any single global uncertainty score.
arXiv:2608.30044v1 Announce Type: new Abstract: Language models are commonly compared by averaging scores across a benchmark list with equal weight. Such lists grow through publication outside an exp...
Large Language Model judges are commonly used to rank texts via pairwise comparison, with reliability traditionally measured by position bias, transitivity, and pairwise agreement. This paper argues that these proxies are misleading because they are dominated by close‑rank‑gap pairs, which contribute little to the overall ranking, while far‑gap pairs carry the true ranking signal. Experiments on simulations and human‑rated corpora show weak correlation between the proxies and actual ranking accuracy, suggesting judges should be evaluated using rank‑gap‑conditional metrics against human rankings.
arXiv:2608. 16643v1 Announce Type: cross Abstract: Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation.
Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation. Error-detection benchmarks are typically constructed by injecting errors into notes, such that each erroneous note has a natural counterpart.
TIDE (Tree-Induced Diagnostic Evaluation) transforms diagnostic evaluation into a behavior-driven decision-tree induction process, enabling fine-grained capability diagnosis of large language models. By recursively building a hierarchical capability tree from benchmark samples and model scores, TIDE identifies nuanced contrasts between models that aggregate metrics miss—such as Claude Opus 4.6 outperforming GPT-5.4 on microeconomic market reasoning while falling behind on contract law reasoning. The method leverages LLM judges to propose splits and uses behavior-based information gain to select them, demonstrating its effectiveness across five models and four benchmarks.
arXiv:2606. 26079v1 Announce Type: cross Abstract: Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines.
The paper introduces an information‑theoretic framework for evaluating knowledge tracing (KT) models that moves beyond aggregate metrics like AUC and accuracy. By applying Context Tree Weighting to item‑response histories, the authors create a causal uncertainty coordinate that separates observable uncertainty from unobserved local irreducible uncertainty (LIU). This approach allows model performance to be assessed across different entropy bands, revealing that modern KT models improve mainly in high‑entropy regions while also highlighting areas where apparent gains may be due to noise.