arXiv AI

JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory

arXiv:2509. 22888v2 Announce Type: replace Abstract: Standard LLM evaluation practices compress diverse abilities into single scores, obscuring their inherently multidimensional nature.

arXiv AI

Jun 10

RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

arXiv:2602. 12424v2 Announce Type: replace-cross Abstract: Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field.

By Ziqian Zhang, Xingjian Hu, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun

llms benchmarks

arXiv AI

Aug 6

Revisiting Generalization Across Difficulty Levels: It's Not So Easy

arXiv:2511. 21692v3 Announce Type: replace-cross Abstract: We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation.

By Yeganeh Kordi, Nihal V. Nayak, Max Zuo, Ilana Nguyen, Stephen H. Bach

llms

arXiv AI

Aug 10

Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding

arXiv:2608. 07353v1 Announce Type: cross Abstract: Understanding concepts is fundamental to generalization.

By Karim Radouane, Jose G Moreno, Lynda Tamine

llms nlp benchmarks

arXiv AI

Jul 28

Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models

arXiv:2601. 02580v2 Announce Type: replace-cross Abstract: Traditional methods for determining assessment item parameters, such as difficulty and discrimination, rely heavily on expensive field testing to collect student performance data for Item Response Theory (IRT) calibration.

By Christopher Ormerod

llms fine-tuning

arXiv Machine Learning

Jun 15

Efficient Rationale-based Retrieval: On-policy Distillation from Generative Rerankers based on JEPA

arXiv:2604. 23336v3 Announce Type: replace-cross Abstract: Unlike traditional fact-based retrieval, rationale-based retrieval typically necessitates cross-encoding of query-document pairs using large language models, incurring substantial computational costs.

By Teng Chen, Sheng Xu, Feixiang Guo, Xiaoyu Wang, Qingqing Gu, Hongyan Li, Luo Ji

llms rag robotics efficiency benchmarks

arXiv AI

Jun 29

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

arXiv:2606. 28186v1 Announce Type: cross Abstract: Predicting human item difficulty is central to educational assessment, where reliable estimates support fairness and effective test construction.

By Chenguang Wang, Ming Li, Xinyue Zeng, Zhuochun Li, Hong Jiao, Tianyi Zhou, Dawei Zhou

llms fine-tuning benchmarks safety

arXiv Machine Learning

Aug 3

Can LLMs Really Understand Item Difficulty Levels? Implications for Automated Item Generation Using LLMs

arXiv:2607. 28634v1 Announce Type: cross Abstract: The estimation of item difficulty plays a key role in both formative assessment and large-scale high-stakes summative assessments.

By Xinyi Wang, Hong Jiao, Ming Li, Sydney Peters, Hanna Choi, Tianyi Zhou, Qingshu Xu

llms rag

arXiv AI

Jul 15

CANDI: Contextual Alignment for Niche Domains Question Answering

arXiv:2607. 11891v1 Announce Type: cross Abstract: The deployment of large language models (LLMs) in specialized domains like medical diagnostics and financial advisory necessitates evaluating capabilities beyond general knowledge.

By Megha Chakraborty, Darssan L. Eswaramoorthi, Het Riteshkumar Shah, Madhur Thareja, Michelle A Ihetu, Harshul Raj Surana, Kaushik Roy, Amit Sheth

llms nlp benchmarks safety

arXiv AI

1d ago

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering

arXiv:2608. 14841v1 Announce Type: new Abstract: Long-document visual question answering (VQA) over documents of tens to hundreds of pages mixing text, tables, charts, and figures typically follows retrieve-then-read pipelines.

By Guanchen Wu, Jiayuan Ding, Subhabrata Mukherjee, Carl Yang

llms nlp multimodal

arXiv AI

Jul 22

Hyperdimensional Probe: Decoding LLM Representations via Vector Symbolic Architectures

arXiv:2509. 25045v3 Announce Type: replace-cross Abstract: Despite their capabilities, Large Language Models (LLMs) remain opaque with limited understanding of their internal representations.

By Marco Bronzini, Carlo Nicolini, Bruno Lepri, Jacopo Staiano, Andrea Passerini

llms rag efficiency safety

arXiv Machine Learning

Jun 8

Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios

arXiv:2606. 06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know.

By Tao Liu, Ye Lu, Ruohua Zhang, Siyu Song, Wentao Liu, Aimin Zhou, Hao Hao

llms agents benchmarks safety

arXiv Machine Learning

Jun 9

Structure-Aware Modeling of Multiple-Choice Questions Improves Automatic Difficulty Estimation

arXiv:2606. 08988v1 Announce Type: cross Abstract: Automatic Question Difficulty Estimation (AQDE) holds growing promise for educational assessment because it has the potential to yield difficulty estimates that are competitive with expert judgment, while helping reduce the time and financial burden associated with pilot administrations and scaling to digital testing contexts.

By Gabriel Ortega, Abelino Jim\'enez, S\'everin Lions, Pablo Dartnell

Related stories