← Back to all news
arXiv Machine Learning By Nguyen Dung Son, Dang Quang Minh, Nguyen Huu Loi, Truong Viet Vu, Nguyen Thai Anh

How Far Can Sub-3B Open Language Models Go in Zero-Shot Essay Scoring on an 8 GB Consumer GPU?

Read the original on arXiv Machine Learning →

The study evaluates zero‑shot essay scoring using sub‑3B open language models that run locally on a single 8 GB consumer GPU. Four instruction‑tuned models (Qwen2.5‑0.5B, 1.5B, 3B and SmolLM2‑1.7B) were tested on all eight ASAP‑AES prompts, comparing rubric‑decomposed versus holistic prompting, different aggregation methods, and trait‑mapping strategies. Results show rubric‑decomposed prompting consistently outperforms holistic prompting, trait‑mapping is sensitive to calibration, and longer essays reduce error, yet the best local configuration (macro QWK 0.388) still falls short of human agreement and a length‑only baseline, suggesting these models are best suited for formative, human‑supervised feedback.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
6d ago

Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

The study evaluates large language model (LLM) graders on two computer‑science exams, testing 171 configurations of closed‑ and open‑weights models. While the best LLM configuration achieved a mean absolute error of 1.64/35—better than the 2.61/35 error between two human graders—its performance was highly sensitive to the prompt. A short "strict grader" preamble caused most open‑weight models to exceed acceptable error thresholds or stop grading entirely, whereas fine‑tuning with a single LoRA adapter restored parity with human graders and reduced sensitivity to harsh prompts.

By Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh, Salman Khan, Naeemullah Khan
llmscomputer-visionfine-tuningbenchmarks
More like this →
arXiv AI
Aug 19

Grading Needs a Rubric, Not Intelligence

Small language models can grade open‑ended exam answers as reliably as much larger models when they use an explicit rubric. In experiments with six cost‑efficient model configurations, the rubric decouples grading from judge intelligence, with answer identity explaining 95.6% of score variance and judge identity only 0.2%. Removing rubric criteria or the official answer collapses reliability and inflates scores, showing the rubric’s essential role.

By Jhen-Ke Lin
llms
More like this →
arXiv AI
Sep 11

Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format

The study compares two ways of obtaining predictions from language models fine‑tuned on customer behavior: scoring answer tokens directly versus generating a written rationale and then scoring the resulting answer. Across 13 model‑domain cells covering four retail tasks, scored readouts consistently rank outcomes more accurately than generated readouts, with an AUC improvement ranging from 1.5 to 14.5 points. The authors also find that a third readout—eliciting a probability before any verdict—improves calibration but only when outcome rates are represented in training, and they recommend using generated rationales for interpretability while relying on scored heads for ranking.

By Touchapon Kraisingkorn, Krittin Pachtrachai, Wachiravit Modecrua
llmsfine-tuning
More like this →