The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Alice is a new large-scale German benchmark for Automatic Short Answer Scoring that focuses on rubric-based evaluation. It contains three subtasks—learning performance (Alice‑LP), knowledge elements (Alice‑KE), and skills (Alice‑SK)—and is designed to assess how well students master underlying concepts and skills rather than just answer questions directly. Experiments show that large language models struggle with zero‑shot scoring of knowledge elements and skills, and that rubric text is especially helpful for Alice‑KE and Alice‑SK.
arXiv:2607. 19219v1 Announce Type: cross Abstract: Large language models (LLMs) have been widely applied to automated essay scoring (AES) and automated feedback generation (AFG).
Large language models (LLMs) have been widely applied to automated essay scoring (AES) and automated feedback generation (AFG). However, existing studies rely primarily on prompt engineering or supervised fine-tuning, while systematic research on reinforcement learning (RL) post-training and automated evaluation of feedback quality remains limited.
The paper proposes a response‑free method for estimating difficulty of reading‑comprehension multiple‑choice items by fine‑tuning a transformer on item wording. It introduces two extensions to a baseline joint‑encoding model: a component‑wise variant that encodes passage, question, and options separately, and a multi‑task variant that adds a question‑answering auxiliary task. Experiments on a corpus of nearly 30,000 items show that both extensions outperform the baseline, especially the multi‑task variant across all metrics and the component‑wise variant in rank ordering, even with limited training data.
The paper investigates what gradient similarity measures in data attribution for large language models. By independently varying task and answer format in supervised fine‑tuning benchmarks, the authors show that gradient alignment is driven by answer format rather than task semantics, with strong alignment for shared formats and none for differing formats. This pattern persists across training stages, model sizes, and families, and is evident in the selections of the LESS data‑selection method, which over‑represents its own answer format.
arXiv:2610.13375v1 Announce Type: new Abstract: Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating bett...