Alice is a new large-scale German benchmark for Automatic Short Answer Scoring that focuses on rubric-based evaluation. It contains three subtasks—learning performance (Alice‑LP), knowledge elements (Alice‑KE), and skills (Alice‑SK)—and is designed to assess how well students master underlying concepts and skills rather than just answer questions directly. Experiments show that large language models struggle with zero‑shot scoring of knowledge elements and skills, and that rubric text is especially helpful for Alice‑KE and Alice‑SK.
arXiv:2610.09660v1 Announce Type: new Abstract: Automatic Short Answer Scoring (ASAS) requires models that can score student responses against question-specific criteria while remaining efficient and...
arXiv:2606. 06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know.
arXiv:2608. 20281v1 Announce Type: cross Abstract: Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time.
arXiv:2510.13935v3 Announce Type: replace-cross Abstract: The facts a language model stores are tied to its parameter count, so small models that fit on edge devices fail on expert problems, which ne...
arXiv:2509. 16780v3 Announce Type: replace-cross Abstract: Large language models (LLMs) show promise as educational aids but often lack alignment with specific course materials.
arXiv:2609.36982v1 Announce Type: cross Abstract: Knowledge concept tagging aims to assign specific concept or topic labels to educational content, which is essential for both educators and learners...
LuxIT is a monolingual instruction‑tuning dataset for Luxembourgish, created by synthesizing instruction‑answer pairs from native texts using the DeepSeek‑R1‑0528 model and a quality‑assurance LLM‑as‑judge process. The resulting 227,507 high‑quality pairs were used to fine‑tune 14 LLMs (≤15 B parameters), yielding an average accuracy increase of +5.37 percentage points on standardized Luxembourgish proficiency exams and improvements in macro‑averaged F1 on nine of the fourteen downstream NLP tasks. These findings demonstrate that synthetic monolingual data can effectively enhance LLM performance in low‑resource languages and reveal the complex relationship between exam performance and practical NLP gains.
arXiv:2608. 06933v1 Announce Type: cross Abstract: Today, we improve models by training and evaluating them on problems at the frontier of their abilities.
arXiv:2608. 15535v1 Announce Type: cross Abstract: We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs).
Large language models are increasingly used to assist with statistical and data science tasks, but current evaluations assume the analysis goal is already defined. This paper formalizes the upstream step of Statistical Problem Formulation into two subtasks—classification of the statistical problem and identification of relevant variables—and introduces StatFormBench, a benchmark comprising 1,013 samples from five statistics textbooks and a data science case library. Across 14 open- and closed‑source LLMs, the best zero‑shot models achieve only 72.0% fine‑grained classification accuracy and 63.2% variable set overlap, with no model consistently excelling in both subtasks and limited gains from enhanced prompting strategies.
arXiv:2609.24650v1 Announce Type: new Abstract: Readability assessment is essential for tailoring texts to intended audiences across educational, healthcare, and information retrieval domains. Howeve...
The paper introduces Jev, a specialized decision model that selects among given options rather than generating text, and benchmarks it against 19 large language models (LLMs) across 13 multiple-choice tasks in knowledge, reasoning, and multilingual domains. Jev performs competitively with frontier LLMs on knowledge and commonsense benchmarks, achieving the top score on MMLU-Redux and ARC-Challenge, and surpasses most representative and all small LLMs except in mathematical word problems where it lags behind. The study demonstrates that a specialized decision model can match general-purpose LLMs on knowledge-based decisions but struggles with multi-step calculations.